WIP - custom migrations when object is created
This commit is contained in:
324
CUSTOM_MIGRATIONS_IMPLEMENTATION.md
Normal file
324
CUSTOM_MIGRATIONS_IMPLEMENTATION.md
Normal file
@@ -0,0 +1,324 @@
|
|||||||
|
# Custom Migrations Implementation
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This implementation adds a database-stored migration system for dynamically created objects. Migrations are recorded in a `custom_migrations` table in each tenant database, allowing them to be replayed or used for environment replication in the future.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Components
|
||||||
|
|
||||||
|
#### 1. CustomMigrationService
|
||||||
|
**Location:** `backend/src/migration/custom-migration.service.ts`
|
||||||
|
|
||||||
|
Handles all migration-related operations:
|
||||||
|
|
||||||
|
- **`generateCreateTableSQL(tableName, fields)`** - Generates SQL for creating object tables with standard fields
|
||||||
|
- **`createMigrationRecord()`** - Stores migration metadata in the database
|
||||||
|
- **`executeMigration()`** - Executes a pending migration and updates its status
|
||||||
|
- **`createAndExecuteMigration()`** - Creates and immediately executes a migration
|
||||||
|
- **`getMigrations()`** - Retrieves migration history with filtering
|
||||||
|
- **`ensureMigrationsTable()`** - Ensures the `custom_migrations` table exists
|
||||||
|
|
||||||
|
#### 2. MigrationModule
|
||||||
|
**Location:** `backend/src/migration/migration.module.ts`
|
||||||
|
|
||||||
|
Provides the CustomMigrationService to other modules.
|
||||||
|
|
||||||
|
#### 3. Updated ObjectService
|
||||||
|
**Location:** `backend/src/object/object.service.ts`
|
||||||
|
|
||||||
|
- Injects CustomMigrationService
|
||||||
|
- Calls `createAndExecuteMigration()` when a new object is created
|
||||||
|
- Generates table creation migrations with standard fields
|
||||||
|
|
||||||
|
### Database Schema
|
||||||
|
|
||||||
|
#### custom_migrations Table
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE custom_migrations (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
tenantId UUID NOT NULL,
|
||||||
|
name VARCHAR(255) NOT NULL,
|
||||||
|
description TEXT,
|
||||||
|
type ENUM('create_table', 'add_column', 'alter_column', 'add_index', 'drop_table', 'custom'),
|
||||||
|
sql TEXT NOT NULL,
|
||||||
|
status ENUM('pending', 'executed', 'failed') DEFAULT 'pending',
|
||||||
|
executedAt TIMESTAMP NULL,
|
||||||
|
error TEXT,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
|
INDEX idx_tenantId (tenantId),
|
||||||
|
INDEX idx_status (status),
|
||||||
|
INDEX idx_created_at (created_at)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Generated Object Tables
|
||||||
|
|
||||||
|
When a new object is created (e.g., "Account"), a table is automatically created with:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE accounts (
|
||||||
|
id VARCHAR(36) PRIMARY KEY,
|
||||||
|
ownerId VARCHAR(36),
|
||||||
|
name VARCHAR(255),
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
|
-- Custom fields added here
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Standard Fields:**
|
||||||
|
- `id` - UUID primary key
|
||||||
|
- `ownerId` - User who owns the record
|
||||||
|
- `name` - Primary name field
|
||||||
|
- `created_at` - Record creation timestamp
|
||||||
|
- `updated_at` - Record update timestamp
|
||||||
|
|
||||||
|
### Field Type Mapping
|
||||||
|
|
||||||
|
Custom fields are mapped to SQL column types:
|
||||||
|
|
||||||
|
| Field Type | SQL Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| TEXT, STRING | VARCHAR(255) | |
|
||||||
|
| LONG_TEXT | TEXT | Large text content |
|
||||||
|
| NUMBER, DECIMAL | DECIMAL(18, 2) | |
|
||||||
|
| INTEGER | INT | |
|
||||||
|
| BOOLEAN | BOOLEAN | Defaults to FALSE |
|
||||||
|
| DATE | DATE | |
|
||||||
|
| DATE_TIME | DATETIME | |
|
||||||
|
| EMAIL | VARCHAR(255) | |
|
||||||
|
| URL | VARCHAR(2048) | |
|
||||||
|
| PHONE | VARCHAR(20) | |
|
||||||
|
| CURRENCY | DECIMAL(18, 2) | |
|
||||||
|
| PERCENT | DECIMAL(5, 2) | |
|
||||||
|
| PICKLIST, MULTI_PICKLIST | VARCHAR(255) | |
|
||||||
|
| LOOKUP, BELONGS_TO | VARCHAR(36) | References foreign record ID |
|
||||||
|
|
||||||
|
## Usage Flow
|
||||||
|
|
||||||
|
### Creating a New Object
|
||||||
|
|
||||||
|
1. **User creates object definition:**
|
||||||
|
```
|
||||||
|
POST /api/objects
|
||||||
|
{
|
||||||
|
"apiName": "Account",
|
||||||
|
"label": "Account",
|
||||||
|
"description": "Customer account records"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **ObjectService.createObjectDefinition() executes:**
|
||||||
|
- Inserts object metadata into `object_definitions` table
|
||||||
|
- Generates create table SQL
|
||||||
|
- Creates migration record with status "pending"
|
||||||
|
- Executes migration immediately
|
||||||
|
- Updates migration status to "executed"
|
||||||
|
- Returns object definition
|
||||||
|
|
||||||
|
3. **Result:**
|
||||||
|
- Object is now ready to use
|
||||||
|
- Table exists in database
|
||||||
|
- Migration history is recorded for future replication
|
||||||
|
|
||||||
|
### Migration Execution Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
createAndExecuteMigration()
|
||||||
|
├── createMigrationRecord()
|
||||||
|
│ └── Insert into custom_migrations (status: pending)
|
||||||
|
└── executeMigration()
|
||||||
|
├── Fetch migration record
|
||||||
|
├── Execute SQL
|
||||||
|
├── Update status: executed
|
||||||
|
└── Return migration record
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
Migrations track execution status and errors:
|
||||||
|
|
||||||
|
- **Status: pending** - Not yet executed
|
||||||
|
- **Status: executed** - Successfully completed
|
||||||
|
- **Status: failed** - Error during execution
|
||||||
|
|
||||||
|
Failed migrations are logged and stored with error details for debugging and retry:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
{
|
||||||
|
id: "uuid",
|
||||||
|
status: "failed",
|
||||||
|
error: "Syntax error in SQL...",
|
||||||
|
executedAt: null,
|
||||||
|
updated_at: "2025-12-24T11:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Future Functionality
|
||||||
|
|
||||||
|
### Sandbox Environment Replication
|
||||||
|
|
||||||
|
Stored migrations enable:
|
||||||
|
|
||||||
|
1. **Cloning production environments** - Replay all migrations in new database
|
||||||
|
2. **Data structure export/import** - Export migrations as SQL files
|
||||||
|
3. **Audit trail** - Complete history of schema changes
|
||||||
|
4. **Rollback capability** - Add down migrations for reverting changes
|
||||||
|
5. **Dependency tracking** - Identify object dependencies from migrations
|
||||||
|
|
||||||
|
### Planned Enhancements
|
||||||
|
|
||||||
|
1. **Add down migrations** - Support undoing schema changes
|
||||||
|
2. **Migration dependencies** - Track which migrations depend on others
|
||||||
|
3. **Batch execution** - Run pending migrations together
|
||||||
|
4. **Version control** - Track migration versions and changes
|
||||||
|
5. **Manual migration creation** - API to create custom migrations
|
||||||
|
6. **Migration status dashboard** - UI to view migration history
|
||||||
|
|
||||||
|
## Integration Points
|
||||||
|
|
||||||
|
### ObjectService
|
||||||
|
|
||||||
|
- Uses `getTenantKnexById()` for tenant database connections
|
||||||
|
- Calls CustomMigrationService after creating object definitions
|
||||||
|
- Handles migration execution errors gracefully (logs but doesn't fail)
|
||||||
|
|
||||||
|
### TenantDatabaseService
|
||||||
|
|
||||||
|
- Provides database connections via `getTenantKnexById()`
|
||||||
|
- Connections are cached with prefix `id:${tenantId}`
|
||||||
|
|
||||||
|
### Module Dependencies
|
||||||
|
|
||||||
|
```
|
||||||
|
ObjectModule
|
||||||
|
├── imports: [TenantModule, MigrationModule]
|
||||||
|
└── providers: [ObjectService, CustomMigrationService, ...]
|
||||||
|
|
||||||
|
MigrationModule
|
||||||
|
├── imports: [TenantModule]
|
||||||
|
└── providers: [CustomMigrationService]
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints (Future)
|
||||||
|
|
||||||
|
While not yet exposed via API, these operations could be added:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Get migration history
|
||||||
|
GET /api/migrations?tenantId=xxx&status=executed
|
||||||
|
|
||||||
|
// Get migration details
|
||||||
|
GET /api/migrations/:id
|
||||||
|
|
||||||
|
// Retry failed migration
|
||||||
|
POST /api/migrations/:id/retry
|
||||||
|
|
||||||
|
// Export migrations as SQL
|
||||||
|
GET /api/migrations/export?tenantId=xxx
|
||||||
|
|
||||||
|
// Create custom migration
|
||||||
|
POST /api/migrations
|
||||||
|
{
|
||||||
|
name: "add_field_to_accounts",
|
||||||
|
description: "Add phone_number field",
|
||||||
|
sql: "ALTER TABLE accounts ADD phone_number VARCHAR(20)"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Manual Testing Steps
|
||||||
|
|
||||||
|
1. **Create a new object:**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3000/api/objects \
|
||||||
|
-H "Authorization: Bearer <token>" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"apiName": "TestObject",
|
||||||
|
"label": "Test Object",
|
||||||
|
"description": "Test object creation"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Verify table was created:**
|
||||||
|
```bash
|
||||||
|
# In tenant database
|
||||||
|
SHOW TABLES LIKE 'test_objects';
|
||||||
|
DESCRIBE test_objects;
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Check migration record:**
|
||||||
|
```bash
|
||||||
|
# In tenant database
|
||||||
|
SELECT * FROM custom_migrations WHERE name LIKE '%test_objects%';
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Create a record in the new object:**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3000/api/test-objects \
|
||||||
|
-H "Authorization: Bearer <token>" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"name": "My Test Record"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Migration Fails with SQL Error
|
||||||
|
|
||||||
|
1. Check `custom_migrations` table for error details:
|
||||||
|
```sql
|
||||||
|
SELECT id, name, error, status FROM custom_migrations
|
||||||
|
WHERE status = 'failed';
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Review the generated SQL in the `sql` column
|
||||||
|
|
||||||
|
3. Common issues:
|
||||||
|
- Duplicate table names
|
||||||
|
- Invalid field names (reserved SQL keywords)
|
||||||
|
- Unsupported field types
|
||||||
|
|
||||||
|
### Table Not Created
|
||||||
|
|
||||||
|
1. Verify `custom_migrations` table exists:
|
||||||
|
```sql
|
||||||
|
SHOW TABLES LIKE 'custom_migrations';
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check object service logs for migration execution errors
|
||||||
|
|
||||||
|
3. Manually retry migration:
|
||||||
|
```typescript
|
||||||
|
const migration = await tenantKnex('custom_migrations')
|
||||||
|
.where({ status: 'failed' })
|
||||||
|
.first();
|
||||||
|
await customMigrationService.executeMigration(tenantKnex, migration.id);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
- **Table creation** is synchronous and happens immediately
|
||||||
|
- **Migrations are cached** in custom_migrations table per tenant
|
||||||
|
- **No file I/O** - all operations use database
|
||||||
|
- **Index creation** optimized with proper indexes on common columns (tenantId, status, created_at)
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- **Per-tenant isolation** - Each tenant's migrations stored separately
|
||||||
|
- **No SQL injection** - Using Knex query builder for all operations
|
||||||
|
- **Access control** - Migrations only created/executed by backend service
|
||||||
|
- **Audit trail** - Complete history of all schema changes
|
||||||
|
|
||||||
|
## Related Files
|
||||||
|
|
||||||
|
- [backend/src/object/object.service.ts](backend/src/object/object.service.ts)
|
||||||
|
- [backend/src/migration/custom-migration.service.ts](backend/src/migration/custom-migration.service.ts)
|
||||||
|
- [backend/src/migration/migration.module.ts](backend/src/migration/migration.module.ts)
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
exports.up = function (knex) {
|
||||||
|
return knex.schema.createTable('custom_migrations', (table) => {
|
||||||
|
table.uuid('id').primary().defaultTo(knex.raw('(UUID())'));
|
||||||
|
table.uuid('tenantId').notNullable();
|
||||||
|
table.string('name', 255).notNullable();
|
||||||
|
table.text('description');
|
||||||
|
table.enum('type', [
|
||||||
|
'create_table',
|
||||||
|
'add_column',
|
||||||
|
'alter_column',
|
||||||
|
'add_index',
|
||||||
|
'drop_table',
|
||||||
|
'custom',
|
||||||
|
]).notNullable();
|
||||||
|
table.text('sql').notNullable();
|
||||||
|
table.enum('status', ['pending', 'executed', 'failed']).defaultTo('pending');
|
||||||
|
table.timestamp('executedAt').nullable();
|
||||||
|
table.text('error').nullable();
|
||||||
|
table.timestamps(true, true);
|
||||||
|
|
||||||
|
table.index(['tenantId']);
|
||||||
|
table.index(['status']);
|
||||||
|
table.index(['created_at']);
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
exports.down = function (knex) {
|
||||||
|
return knex.schema.dropTableIfExists('custom_migrations');
|
||||||
|
};
|
||||||
@@ -43,8 +43,9 @@ function decryptPassword(encryptedPassword: string): string {
|
|||||||
function createTenantKnexConnection(tenant: any): Knex {
|
function createTenantKnexConnection(tenant: any): Knex {
|
||||||
const decryptedPassword = decryptPassword(tenant.dbPassword);
|
const decryptedPassword = decryptPassword(tenant.dbPassword);
|
||||||
|
|
||||||
// Replace 'db' hostname with 'localhost' when running outside Docker
|
// Use Docker hostname 'db' when running inside container
|
||||||
const dbHost = tenant.dbHost === 'db' ? 'localhost' : tenant.dbHost;
|
// The dbHost will be 'db' for Docker connections or 'localhost' for local development
|
||||||
|
const dbHost = tenant.dbHost;
|
||||||
|
|
||||||
return knex({
|
return knex({
|
||||||
client: 'mysql2',
|
client: 'mysql2',
|
||||||
@@ -82,7 +83,7 @@ async function migrateTenant(tenant: any): Promise<void> {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error(`❌ ${tenant.name}: Migration failed:`, error.message);
|
console.error(`❌ ${tenant.name}: Migration failed:`, error);
|
||||||
throw error;
|
throw error;
|
||||||
} finally {
|
} finally {
|
||||||
await tenantKnex.destroy();
|
await tenantKnex.destroy();
|
||||||
|
|||||||
306
backend/src/migration/custom-migration.service.ts
Normal file
306
backend/src/migration/custom-migration.service.ts
Normal file
@@ -0,0 +1,306 @@
|
|||||||
|
import { Injectable, Logger } from '@nestjs/common';
|
||||||
|
import { Knex } from 'knex';
|
||||||
|
|
||||||
|
export interface CustomMigrationRecord {
|
||||||
|
id: string;
|
||||||
|
tenantId: string;
|
||||||
|
name: string;
|
||||||
|
description: string;
|
||||||
|
type: 'create_table' | 'add_column' | 'alter_column' | 'add_index' | 'drop_table' | 'custom';
|
||||||
|
sql: string;
|
||||||
|
status: 'pending' | 'executed' | 'failed';
|
||||||
|
executedAt?: Date;
|
||||||
|
error?: string;
|
||||||
|
createdAt: Date;
|
||||||
|
updatedAt: Date;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Injectable()
|
||||||
|
export class CustomMigrationService {
|
||||||
|
private readonly logger = new Logger(CustomMigrationService.name);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generate SQL to create a table with standard fields
|
||||||
|
*/
|
||||||
|
generateCreateTableSQL(
|
||||||
|
tableName: string,
|
||||||
|
fields: {
|
||||||
|
apiName: string;
|
||||||
|
type: string;
|
||||||
|
isRequired?: boolean;
|
||||||
|
isUnique?: boolean;
|
||||||
|
defaultValue?: string;
|
||||||
|
}[] = [],
|
||||||
|
): string {
|
||||||
|
// Start with standard fields
|
||||||
|
const columns: string[] = [
|
||||||
|
'`id` VARCHAR(36) PRIMARY KEY',
|
||||||
|
'`ownerId` VARCHAR(36)',
|
||||||
|
'`name` VARCHAR(255)',
|
||||||
|
'`created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP',
|
||||||
|
'`updated_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP',
|
||||||
|
];
|
||||||
|
|
||||||
|
// Add custom fields
|
||||||
|
for (const field of fields) {
|
||||||
|
const column = this.fieldToColumn(field);
|
||||||
|
columns.push(column);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add foreign key and index for ownerId
|
||||||
|
columns.push('INDEX `idx_owner` (`ownerId`)');
|
||||||
|
|
||||||
|
return `CREATE TABLE IF NOT EXISTS \`${tableName}\` (
|
||||||
|
${columns.join(',\n ')}
|
||||||
|
)`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert field definition to SQL column definition
|
||||||
|
*/
|
||||||
|
private fieldToColumn(field: {
|
||||||
|
apiName: string;
|
||||||
|
type: string;
|
||||||
|
isRequired?: boolean;
|
||||||
|
isUnique?: boolean;
|
||||||
|
defaultValue?: string;
|
||||||
|
}): string {
|
||||||
|
const columnName = field.apiName;
|
||||||
|
let columnDef = `\`${columnName}\``;
|
||||||
|
|
||||||
|
// Map field types to SQL types
|
||||||
|
switch (field.type.toUpperCase()) {
|
||||||
|
case 'TEXT':
|
||||||
|
case 'STRING':
|
||||||
|
columnDef += ' VARCHAR(255)';
|
||||||
|
break;
|
||||||
|
case 'LONG_TEXT':
|
||||||
|
columnDef += ' LONGTEXT';
|
||||||
|
break;
|
||||||
|
case 'NUMBER':
|
||||||
|
case 'DECIMAL':
|
||||||
|
columnDef += ' DECIMAL(18, 2)';
|
||||||
|
break;
|
||||||
|
case 'INTEGER':
|
||||||
|
columnDef += ' INT';
|
||||||
|
break;
|
||||||
|
case 'BOOLEAN':
|
||||||
|
columnDef += ' BOOLEAN DEFAULT FALSE';
|
||||||
|
break;
|
||||||
|
case 'DATE':
|
||||||
|
columnDef += ' DATE';
|
||||||
|
break;
|
||||||
|
case 'DATE_TIME':
|
||||||
|
columnDef += ' DATETIME';
|
||||||
|
break;
|
||||||
|
case 'EMAIL':
|
||||||
|
columnDef += ' VARCHAR(255)';
|
||||||
|
break;
|
||||||
|
case 'URL':
|
||||||
|
columnDef += ' VARCHAR(2048)';
|
||||||
|
break;
|
||||||
|
case 'PHONE':
|
||||||
|
columnDef += ' VARCHAR(20)';
|
||||||
|
break;
|
||||||
|
case 'CURRENCY':
|
||||||
|
columnDef += ' DECIMAL(18, 2)';
|
||||||
|
break;
|
||||||
|
case 'PERCENT':
|
||||||
|
columnDef += ' DECIMAL(5, 2)';
|
||||||
|
break;
|
||||||
|
case 'PICKLIST':
|
||||||
|
case 'MULTI_PICKLIST':
|
||||||
|
columnDef += ' VARCHAR(255)';
|
||||||
|
break;
|
||||||
|
case 'LOOKUP':
|
||||||
|
case 'BELONGS_TO':
|
||||||
|
columnDef += ' VARCHAR(36)';
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
columnDef += ' VARCHAR(255)';
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add constraints
|
||||||
|
if (field.isRequired) {
|
||||||
|
columnDef += ' NOT NULL';
|
||||||
|
} else {
|
||||||
|
columnDef += ' NULL';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (field.isUnique) {
|
||||||
|
columnDef += ' UNIQUE';
|
||||||
|
}
|
||||||
|
|
||||||
|
if (field.defaultValue !== undefined && field.defaultValue !== null) {
|
||||||
|
columnDef += ` DEFAULT '${field.defaultValue}'`;
|
||||||
|
}
|
||||||
|
|
||||||
|
return columnDef;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create a custom migration record in the database
|
||||||
|
*/
|
||||||
|
async createMigrationRecord(
|
||||||
|
tenantKnex: Knex,
|
||||||
|
data: {
|
||||||
|
tenantId: string;
|
||||||
|
name: string;
|
||||||
|
description: string;
|
||||||
|
type: 'create_table' | 'add_column' | 'alter_column' | 'add_index' | 'drop_table' | 'custom';
|
||||||
|
sql: string;
|
||||||
|
},
|
||||||
|
): Promise<CustomMigrationRecord> {
|
||||||
|
// Ensure custom_migrations table exists
|
||||||
|
await this.ensureMigrationsTable(tenantKnex);
|
||||||
|
|
||||||
|
const id = require('crypto').randomUUID();
|
||||||
|
const now = new Date();
|
||||||
|
|
||||||
|
await tenantKnex('custom_migrations').insert({
|
||||||
|
id,
|
||||||
|
tenantId: data.tenantId,
|
||||||
|
name: data.name,
|
||||||
|
description: data.description,
|
||||||
|
type: data.type,
|
||||||
|
sql: data.sql,
|
||||||
|
status: 'pending',
|
||||||
|
created_at: now,
|
||||||
|
updated_at: now,
|
||||||
|
});
|
||||||
|
|
||||||
|
return tenantKnex('custom_migrations').where({ id }).first();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Execute a pending migration and update its status
|
||||||
|
*/
|
||||||
|
async executeMigration(
|
||||||
|
tenantKnex: Knex,
|
||||||
|
migrationId: string,
|
||||||
|
): Promise<CustomMigrationRecord> {
|
||||||
|
try {
|
||||||
|
// Get the migration record
|
||||||
|
const migration = await tenantKnex('custom_migrations')
|
||||||
|
.where({ id: migrationId })
|
||||||
|
.first();
|
||||||
|
|
||||||
|
if (!migration) {
|
||||||
|
throw new Error(`Migration ${migrationId} not found`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (migration.status === 'executed') {
|
||||||
|
this.logger.log(`Migration ${migrationId} already executed`);
|
||||||
|
return migration;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Execute the SQL
|
||||||
|
this.logger.log(`Executing migration: ${migration.name}`);
|
||||||
|
await tenantKnex.raw(migration.sql);
|
||||||
|
|
||||||
|
// Update status
|
||||||
|
const now = new Date();
|
||||||
|
await tenantKnex('custom_migrations')
|
||||||
|
.where({ id: migrationId })
|
||||||
|
.update({
|
||||||
|
status: 'executed',
|
||||||
|
executedAt: now,
|
||||||
|
updated_at: now,
|
||||||
|
});
|
||||||
|
|
||||||
|
this.logger.log(`Migration ${migration.name} executed successfully`);
|
||||||
|
return tenantKnex('custom_migrations').where({ id: migrationId }).first();
|
||||||
|
} catch (error) {
|
||||||
|
this.logger.error(`Failed to execute migration ${migrationId}:`, error);
|
||||||
|
|
||||||
|
// Update status with error
|
||||||
|
const now = new Date();
|
||||||
|
await tenantKnex('custom_migrations')
|
||||||
|
.where({ id: migrationId })
|
||||||
|
.update({
|
||||||
|
status: 'failed',
|
||||||
|
error: error.message,
|
||||||
|
updated_at: now,
|
||||||
|
});
|
||||||
|
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create and execute a migration in one step
|
||||||
|
*/
|
||||||
|
async createAndExecuteMigration(
|
||||||
|
tenantKnex: Knex,
|
||||||
|
tenantId: string,
|
||||||
|
data: {
|
||||||
|
name: string;
|
||||||
|
description: string;
|
||||||
|
type: 'create_table' | 'add_column' | 'alter_column' | 'add_index' | 'drop_table' | 'custom';
|
||||||
|
sql: string;
|
||||||
|
},
|
||||||
|
): Promise<CustomMigrationRecord> {
|
||||||
|
// Create the migration record
|
||||||
|
const migration = await this.createMigrationRecord(tenantKnex, {
|
||||||
|
tenantId,
|
||||||
|
...data,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Execute it immediately
|
||||||
|
return this.executeMigration(tenantKnex, migration.id);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Ensure the custom_migrations table exists in the tenant database
|
||||||
|
*/
|
||||||
|
private async ensureMigrationsTable(tenantKnex: Knex): Promise<void> {
|
||||||
|
const hasTable = await tenantKnex.schema.hasTable('custom_migrations');
|
||||||
|
|
||||||
|
if (!hasTable) {
|
||||||
|
await tenantKnex.schema.createTable('custom_migrations', (table) => {
|
||||||
|
table.uuid('id').primary();
|
||||||
|
table.uuid('tenantId').notNullable();
|
||||||
|
table.string('name', 255).notNullable();
|
||||||
|
table.text('description');
|
||||||
|
table.enum('type', ['create_table', 'add_column', 'alter_column', 'add_index', 'drop_table', 'custom']).notNullable();
|
||||||
|
table.text('sql').notNullable();
|
||||||
|
table.enum('status', ['pending', 'executed', 'failed']).defaultTo('pending');
|
||||||
|
table.timestamp('executedAt').nullable();
|
||||||
|
table.text('error').nullable();
|
||||||
|
table.timestamps(true, true);
|
||||||
|
|
||||||
|
table.index(['tenantId']);
|
||||||
|
table.index(['status']);
|
||||||
|
table.index(['created_at']);
|
||||||
|
});
|
||||||
|
|
||||||
|
this.logger.log('Created custom_migrations table');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get all migrations for a tenant
|
||||||
|
*/
|
||||||
|
async getMigrations(
|
||||||
|
tenantKnex: Knex,
|
||||||
|
tenantId: string,
|
||||||
|
filter?: {
|
||||||
|
status?: 'pending' | 'executed' | 'failed';
|
||||||
|
type?: string;
|
||||||
|
},
|
||||||
|
): Promise<CustomMigrationRecord[]> {
|
||||||
|
await this.ensureMigrationsTable(tenantKnex);
|
||||||
|
|
||||||
|
let query = tenantKnex('custom_migrations').where({ tenantId });
|
||||||
|
|
||||||
|
if (filter?.status) {
|
||||||
|
query = query.where({ status: filter.status });
|
||||||
|
}
|
||||||
|
|
||||||
|
if (filter?.type) {
|
||||||
|
query = query.where({ type: filter.type });
|
||||||
|
}
|
||||||
|
|
||||||
|
return query.orderBy('created_at', 'asc');
|
||||||
|
}
|
||||||
|
}
|
||||||
10
backend/src/migration/migration.module.ts
Normal file
10
backend/src/migration/migration.module.ts
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { CustomMigrationService } from './custom-migration.service';
|
||||||
|
import { TenantModule } from '../tenant/tenant.module';
|
||||||
|
|
||||||
|
@Module({
|
||||||
|
imports: [TenantModule],
|
||||||
|
providers: [CustomMigrationService],
|
||||||
|
exports: [CustomMigrationService],
|
||||||
|
})
|
||||||
|
export class MigrationModule {}
|
||||||
@@ -5,9 +5,10 @@ import { SetupObjectController } from './setup-object.controller';
|
|||||||
import { SchemaManagementService } from './schema-management.service';
|
import { SchemaManagementService } from './schema-management.service';
|
||||||
import { FieldMapperService } from './field-mapper.service';
|
import { FieldMapperService } from './field-mapper.service';
|
||||||
import { TenantModule } from '../tenant/tenant.module';
|
import { TenantModule } from '../tenant/tenant.module';
|
||||||
|
import { MigrationModule } from '../migration/migration.module';
|
||||||
|
|
||||||
@Module({
|
@Module({
|
||||||
imports: [TenantModule],
|
imports: [TenantModule, MigrationModule],
|
||||||
providers: [ObjectService, SchemaManagementService, FieldMapperService],
|
providers: [ObjectService, SchemaManagementService, FieldMapperService],
|
||||||
controllers: [RuntimeObjectController, SetupObjectController],
|
controllers: [RuntimeObjectController, SetupObjectController],
|
||||||
exports: [ObjectService, SchemaManagementService, FieldMapperService],
|
exports: [ObjectService, SchemaManagementService, FieldMapperService],
|
||||||
|
|||||||
@@ -1,13 +1,18 @@
|
|||||||
import { Injectable, NotFoundException } from '@nestjs/common';
|
import { Injectable, NotFoundException } from '@nestjs/common';
|
||||||
import { TenantDatabaseService } from '../tenant/tenant-database.service';
|
import { TenantDatabaseService } from '../tenant/tenant-database.service';
|
||||||
|
import { CustomMigrationService } from '../migration/custom-migration.service';
|
||||||
|
|
||||||
@Injectable()
|
@Injectable()
|
||||||
export class ObjectService {
|
export class ObjectService {
|
||||||
constructor(private tenantDbService: TenantDatabaseService) {}
|
constructor(
|
||||||
|
private tenantDbService: TenantDatabaseService,
|
||||||
|
private customMigrationService: CustomMigrationService,
|
||||||
|
) {}
|
||||||
|
|
||||||
// Setup endpoints - Object metadata management
|
// Setup endpoints - Object metadata management
|
||||||
async getObjectDefinitions(tenantId: string) {
|
async getObjectDefinitions(tenantId: string) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
const objects = await knex('object_definitions')
|
const objects = await knex('object_definitions')
|
||||||
.select('object_definitions.*')
|
.select('object_definitions.*')
|
||||||
@@ -28,7 +33,8 @@ export class ObjectService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async getObjectDefinition(tenantId: string, apiName: string) {
|
async getObjectDefinition(tenantId: string, apiName: string) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
const obj = await knex('object_definitions')
|
const obj = await knex('object_definitions')
|
||||||
.where({ apiName })
|
.where({ apiName })
|
||||||
@@ -69,15 +75,104 @@ export class ObjectService {
|
|||||||
isSystem?: boolean;
|
isSystem?: boolean;
|
||||||
},
|
},
|
||||||
) {
|
) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
// Resolve tenant ID in case a slug was passed
|
||||||
const [id] = await knex('object_definitions').insert({
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
id: knex.raw('(UUID())'),
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
|
// Generate UUID for the new object
|
||||||
|
const objectId = require('crypto').randomUUID();
|
||||||
|
|
||||||
|
// Create the object definition record
|
||||||
|
await knex('object_definitions').insert({
|
||||||
|
id: objectId,
|
||||||
...data,
|
...data,
|
||||||
created_at: knex.fn.now(),
|
created_at: knex.fn.now(),
|
||||||
updated_at: knex.fn.now(),
|
updated_at: knex.fn.now(),
|
||||||
});
|
});
|
||||||
|
|
||||||
return knex('object_definitions').where({ id }).first();
|
const objectDef = await knex('object_definitions').where({ id: objectId }).first();
|
||||||
|
|
||||||
|
// Create standard field definitions (only if they don't already exist)
|
||||||
|
const standardFields = [
|
||||||
|
{
|
||||||
|
apiName: 'ownerId',
|
||||||
|
label: 'Owner',
|
||||||
|
type: 'LOOKUP',
|
||||||
|
description: 'The user who owns this record',
|
||||||
|
isRequired: true,
|
||||||
|
isUnique: false,
|
||||||
|
referenceObject: null,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
apiName: 'name',
|
||||||
|
label: 'Name',
|
||||||
|
type: 'TEXT',
|
||||||
|
description: 'The primary name field for this record',
|
||||||
|
isRequired: true,
|
||||||
|
isUnique: false,
|
||||||
|
referenceObject: null,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
apiName: 'created_at',
|
||||||
|
label: 'Created At',
|
||||||
|
type: 'DATE_TIME',
|
||||||
|
description: 'The timestamp when this record was created',
|
||||||
|
isRequired: true,
|
||||||
|
isUnique: false,
|
||||||
|
referenceObject: null,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
apiName: 'updated_at',
|
||||||
|
label: 'Updated At',
|
||||||
|
type: 'DATE_TIME',
|
||||||
|
description: 'The timestamp when this record was last updated',
|
||||||
|
isRequired: true,
|
||||||
|
isUnique: false,
|
||||||
|
referenceObject: null,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
// Insert standard field definitions that don't already exist
|
||||||
|
for (const field of standardFields) {
|
||||||
|
const existingField = await knex('field_definitions')
|
||||||
|
.where({
|
||||||
|
objectDefinitionId: objectDef.id,
|
||||||
|
apiName: field.apiName,
|
||||||
|
})
|
||||||
|
.first();
|
||||||
|
|
||||||
|
if (!existingField) {
|
||||||
|
await knex('field_definitions').insert({
|
||||||
|
id: knex.raw('(UUID())'),
|
||||||
|
objectDefinitionId: objectDef.id,
|
||||||
|
...field,
|
||||||
|
created_at: knex.fn.now(),
|
||||||
|
updated_at: knex.fn.now(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create a migration to create the table
|
||||||
|
const tableName = this.getTableName(data.apiName);
|
||||||
|
const createTableSQL = this.customMigrationService.generateCreateTableSQL(tableName);
|
||||||
|
|
||||||
|
try {
|
||||||
|
await this.customMigrationService.createAndExecuteMigration(
|
||||||
|
knex,
|
||||||
|
resolvedTenantId,
|
||||||
|
{
|
||||||
|
name: `create_${tableName}_table`,
|
||||||
|
description: `Create table for ${data.label} object`,
|
||||||
|
type: 'create_table',
|
||||||
|
sql: createTableSQL,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
// Log the error but don't fail - migration is recorded for future retry
|
||||||
|
console.error(`Failed to execute table creation migration: ${error.message}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return objectDef;
|
||||||
}
|
}
|
||||||
|
|
||||||
async createFieldDefinition(
|
async createFieldDefinition(
|
||||||
@@ -94,7 +189,8 @@ export class ObjectService {
|
|||||||
defaultValue?: string;
|
defaultValue?: string;
|
||||||
},
|
},
|
||||||
) {
|
) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
const obj = await this.getObjectDefinition(tenantId, objectApiName);
|
const obj = await this.getObjectDefinition(tenantId, objectApiName);
|
||||||
|
|
||||||
const [id] = await knex('field_definitions').insert({
|
const [id] = await knex('field_definitions').insert({
|
||||||
@@ -134,7 +230,8 @@ export class ObjectService {
|
|||||||
userId: string,
|
userId: string,
|
||||||
filters?: any,
|
filters?: any,
|
||||||
) {
|
) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
// Verify object exists
|
// Verify object exists
|
||||||
await this.getObjectDefinition(tenantId, objectApiName);
|
await this.getObjectDefinition(tenantId, objectApiName);
|
||||||
@@ -163,7 +260,8 @@ export class ObjectService {
|
|||||||
recordId: string,
|
recordId: string,
|
||||||
userId: string,
|
userId: string,
|
||||||
) {
|
) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
// Verify object exists
|
// Verify object exists
|
||||||
await this.getObjectDefinition(tenantId, objectApiName);
|
await this.getObjectDefinition(tenantId, objectApiName);
|
||||||
@@ -193,7 +291,8 @@ export class ObjectService {
|
|||||||
data: any,
|
data: any,
|
||||||
userId: string,
|
userId: string,
|
||||||
) {
|
) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
// Verify object exists
|
// Verify object exists
|
||||||
await this.getObjectDefinition(tenantId, objectApiName);
|
await this.getObjectDefinition(tenantId, objectApiName);
|
||||||
@@ -226,7 +325,8 @@ export class ObjectService {
|
|||||||
data: any,
|
data: any,
|
||||||
userId: string,
|
userId: string,
|
||||||
) {
|
) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
// Verify object exists and user has access
|
// Verify object exists and user has access
|
||||||
await this.getRecord(tenantId, objectApiName, recordId, userId);
|
await this.getRecord(tenantId, objectApiName, recordId, userId);
|
||||||
@@ -246,7 +346,8 @@ export class ObjectService {
|
|||||||
recordId: string,
|
recordId: string,
|
||||||
userId: string,
|
userId: string,
|
||||||
) {
|
) {
|
||||||
const knex = await this.tenantDbService.getTenantKnex(tenantId);
|
const resolvedTenantId = await this.tenantDbService.resolveTenantId(tenantId);
|
||||||
|
const knex = await this.tenantDbService.getTenantKnexById(resolvedTenantId);
|
||||||
|
|
||||||
// Verify object exists and user has access
|
// Verify object exists and user has access
|
||||||
await this.getRecord(tenantId, objectApiName, recordId, userId);
|
await this.getRecord(tenantId, objectApiName, recordId, userId);
|
||||||
|
|||||||
@@ -169,6 +169,36 @@ export class TenantDatabaseService {
|
|||||||
return domainRecord.tenant;
|
return domainRecord.tenant;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Resolve tenant by ID or slug
|
||||||
|
* Tries ID first, then falls back to slug
|
||||||
|
*/
|
||||||
|
async resolveTenantId(idOrSlug: string): Promise<string> {
|
||||||
|
const centralPrisma = getCentralPrisma();
|
||||||
|
|
||||||
|
// Try by ID first
|
||||||
|
let tenant = await centralPrisma.tenant.findUnique({
|
||||||
|
where: { id: idOrSlug },
|
||||||
|
});
|
||||||
|
|
||||||
|
// If not found, try by slug
|
||||||
|
if (!tenant) {
|
||||||
|
tenant = await centralPrisma.tenant.findUnique({
|
||||||
|
where: { slug: idOrSlug },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!tenant) {
|
||||||
|
throw new Error(`Tenant ${idOrSlug} not found`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (tenant.status !== 'active') {
|
||||||
|
throw new Error(`Tenant ${tenant.name} is not active`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return tenant.id;
|
||||||
|
}
|
||||||
|
|
||||||
async disconnectTenant(tenantId: string) {
|
async disconnectTenant(tenantId: string) {
|
||||||
const connection = this.tenantConnections.get(tenantId);
|
const connection = this.tenantConnections.get(tenantId);
|
||||||
if (connection) {
|
if (connection) {
|
||||||
|
|||||||
Reference in New Issue
Block a user