mirror of https://github.com/Budibase/budibase.git
Rory Powell
4 years ago
87 changed files with 2051 additions and 1072 deletions
@ -0,0 +1,93 @@ |
|||
|
|||
# Budibase CI Pipelines |
|||
|
|||
Welcome to the budibase CI pipelines directory. This document details what each of the CI pipelines are for, and come common combinations. |
|||
|
|||
## All CI Pipelines |
|||
|
|||
### Note |
|||
- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. |
|||
|
|||
### Standard CI Build Job (budibase_ci.yml) |
|||
Triggers: |
|||
- PR or push to develop |
|||
- PR or push to master |
|||
|
|||
The standard CI Build job is what runs when you raise a PR to develop or master. |
|||
- Installs all dependencies, |
|||
- builds the project |
|||
- run the unit tests |
|||
- Generate test coverage metrics with codecov |
|||
- Run the cypress tests |
|||
|
|||
### Release Develop Job (release-develop.yml) |
|||
Triggers: |
|||
- Push to develop |
|||
|
|||
The job responsible for building, tagging and pushing docker images out to the test and staging environments. |
|||
- Installs all dependencies |
|||
- builds the project |
|||
- run the unit tests |
|||
- publish the budibase JS packages under a prerelease tag to NPM |
|||
- build, tag and push docker images under the `develop` tag to docker hub |
|||
|
|||
These images will then be pulled by the test and staging environments, updating the latest automatically. Discord notifications are sent to the #infra channel when this occurs. |
|||
|
|||
### Release Job (release.yml) |
|||
Triggers: |
|||
- Push to master |
|||
|
|||
This job is responsible for building and pushing the latest code to NPM and docker hub, so that it can be deployed. |
|||
- Installs all dependencies |
|||
- builds the project |
|||
- run the unit tests |
|||
- publish the budibase JS packages under a release tag to NPM (always incremented by patch versions) |
|||
- build, tag and push docker images under the `v.x.x.x` (the tag of the NPM release) tag to docker hub |
|||
|
|||
### Release Selfhost Job (release-selfhost.yml) |
|||
Triggers: |
|||
- Manual Workflow Dispatch Trigger |
|||
|
|||
This job is responsible for delivering the latest version of budibase to those that are self-hosting. |
|||
|
|||
This job relies on the release job to have run first, so the latest image is pushed to dockerhub. This job then will pull the latest version from `lerna.json` and try to find an image in dockerhub corresponding to that version. For example, if the version in `lerna.json` is `1.0.0`: |
|||
- Pull the images for all budibase services tagged `v1.0.0` from dockerhub |
|||
- Tag these images as `latest` |
|||
- Push them back to dockerhub. This now means anyone who pulls `latest` (self hosters using docker-compose) will get the latest version. |
|||
- Build and release the budibase helm chart for kubernetes users |
|||
- Perform a github release with the latest version. You can see previous releases here (https://github.com/Budibase/budibase/releases) |
|||
|
|||
|
|||
### Cloud Deploy (deploy-cloud.yml) |
|||
Triggers: |
|||
- Manual Workflow Dispatch Trigger |
|||
|
|||
This job is responsible for deploying to our production, cloud kubernetes environment. You must run the release job first, to ensure that the latest images have been built and pushed to docker hub. You can also manually enter a version number for this job, so you can perform rollbacks or upgrade to a specific version. After kicking off this job, the following will occur: |
|||
|
|||
- Checks out the master branch |
|||
- Pulls the latest `values.yaml` from budibase infra, a private repo containing budibases infrastructure configuration |
|||
- Gets the latest budibase version from `lerna.json`, if it hasn't been specified in the workflow when you kicked it off |
|||
- Configures AWS Credentials |
|||
- Deploys the helm chart in the budibase repo to our production EKS cluster, injecting the `values.yaml` we pulled from budibase-infra |
|||
- Fires off a discord webhook in the #infra channel to show that the deployment completely successfully. |
|||
|
|||
## Common Workflows |
|||
|
|||
### Deploy Changes to Production (Release) |
|||
- Merge `develop` into `master` |
|||
- Wait for budibase CI job and release job to run |
|||
- Run cloud deploy job |
|||
- Run release selfhost job |
|||
|
|||
### Deploy Changes to Production (Hotfix) |
|||
- Branch off `master` |
|||
- Perform your hotfix |
|||
- Merge back into `master` |
|||
- Wait for budibase CI job and release job to run |
|||
- Run cloud deploy job |
|||
- Run release selfhost job |
|||
|
|||
### Rollback A Bad Cloud Deployment |
|||
- Kick off cloud deploy job |
|||
- Ensure you are running off master |
|||
- Enter the version number of the last known good version of budibase. For example `1.0.0` |
|||
@ -0,0 +1,9 @@ |
|||
dependencies: |
|||
- name: couchdb |
|||
repository: https://apache.github.io/couchdb-helm |
|||
version: 3.3.4 |
|||
- name: ingress-nginx |
|||
repository: https://kubernetes.github.io/ingress-nginx |
|||
version: 4.0.13 |
|||
digest: sha256:20892705c2d8e64c98257d181063a514ac55013e2b43399a6e54868a97f97845 |
|||
generated: "2021-12-30T18:55:30.878411Z" |
|||
Binary file not shown.
Binary file not shown.
@ -0,0 +1 @@ |
|||
module.exports = require("./src/auth") |
|||
@ -0,0 +1 @@ |
|||
module.exports = require("./src/middleware") |
|||
@ -0,0 +1,4 @@ |
|||
module.exports = { |
|||
...require("./src/objectStore"), |
|||
...require("./src/objectStore/utils"), |
|||
} |
|||
@ -0,0 +1,45 @@ |
|||
const passport = require("koa-passport") |
|||
const LocalStrategy = require("passport-local").Strategy |
|||
const JwtStrategy = require("passport-jwt").Strategy |
|||
const { getGlobalDB } = require("./tenancy") |
|||
const { |
|||
jwt, |
|||
local, |
|||
authenticated, |
|||
google, |
|||
oidc, |
|||
auditLog, |
|||
tenancy, |
|||
appTenancy, |
|||
authError, |
|||
} = require("./middleware") |
|||
|
|||
// Strategies
|
|||
passport.use(new LocalStrategy(local.options, local.authenticate)) |
|||
passport.use(new JwtStrategy(jwt.options, jwt.authenticate)) |
|||
|
|||
passport.serializeUser((user, done) => done(null, user)) |
|||
|
|||
passport.deserializeUser(async (user, done) => { |
|||
const db = getGlobalDB() |
|||
|
|||
try { |
|||
const user = await db.get(user._id) |
|||
return done(null, user) |
|||
} catch (err) { |
|||
console.error("User not found", err) |
|||
return done(null, false, { message: "User not found" }) |
|||
} |
|||
}) |
|||
|
|||
module.exports = { |
|||
buildAuthMiddleware: authenticated, |
|||
passport, |
|||
google, |
|||
oidc, |
|||
jwt: require("jsonwebtoken"), |
|||
buildTenancyMiddleware: tenancy, |
|||
buildAppTenancyMiddleware: appTenancy, |
|||
auditLog, |
|||
authError, |
|||
} |
|||
@ -1,71 +1,17 @@ |
|||
const passport = require("koa-passport") |
|||
const LocalStrategy = require("passport-local").Strategy |
|||
const JwtStrategy = require("passport-jwt").Strategy |
|||
const { StaticDatabases } = require("./db/utils") |
|||
const { getGlobalDB } = require("./tenancy") |
|||
const { |
|||
jwt, |
|||
local, |
|||
authenticated, |
|||
google, |
|||
oidc, |
|||
auditLog, |
|||
tenancy, |
|||
appTenancy, |
|||
authError, |
|||
} = require("./middleware") |
|||
const { setDB } = require("./db") |
|||
const userCache = require("./cache/user") |
|||
|
|||
// Strategies
|
|||
passport.use(new LocalStrategy(local.options, local.authenticate)) |
|||
passport.use(new JwtStrategy(jwt.options, jwt.authenticate)) |
|||
|
|||
passport.serializeUser((user, done) => done(null, user)) |
|||
|
|||
passport.deserializeUser(async (user, done) => { |
|||
const db = getGlobalDB() |
|||
|
|||
try { |
|||
const user = await db.get(user._id) |
|||
return done(null, user) |
|||
} catch (err) { |
|||
console.error("User not found", err) |
|||
return done(null, false, { message: "User not found" }) |
|||
} |
|||
}) |
|||
|
|||
module.exports = { |
|||
init(pouch) { |
|||
setDB(pouch) |
|||
}, |
|||
db: require("./db/utils"), |
|||
redis: { |
|||
Client: require("./redis"), |
|||
utils: require("./redis/utils"), |
|||
}, |
|||
objectStore: { |
|||
...require("./objectStore"), |
|||
...require("./objectStore/utils"), |
|||
}, |
|||
utils: { |
|||
...require("./utils"), |
|||
...require("./hashing"), |
|||
}, |
|||
auth: { |
|||
buildAuthMiddleware: authenticated, |
|||
passport, |
|||
google, |
|||
oidc, |
|||
jwt: require("jsonwebtoken"), |
|||
buildTenancyMiddleware: tenancy, |
|||
buildAppTenancyMiddleware: appTenancy, |
|||
auditLog, |
|||
authError, |
|||
}, |
|||
cache: { |
|||
user: userCache, |
|||
}, |
|||
StaticDatabases, |
|||
constants: require("./constants"), |
|||
// some default exports from the library, however these ideally shouldn't
|
|||
// be used, instead the syntax require("@budibase/backend-core/db") should be used
|
|||
StaticDatabases: require("./db/utils").StaticDatabases, |
|||
db: require("../db"), |
|||
redis: require("../redis"), |
|||
objectStore: require("../objectStore"), |
|||
utils: require("../utils"), |
|||
cache: require("../cache"), |
|||
auth: require("../auth"), |
|||
constants: require("../constants"), |
|||
} |
|||
|
|||
@ -0,0 +1,4 @@ |
|||
module.exports = { |
|||
...require("./src/utils"), |
|||
...require("./src/hashing"), |
|||
} |
|||
@ -1,133 +0,0 @@ |
|||
import { cloneDeep } from "lodash/fp" |
|||
import { fetchTableData, fetchTableDefinition } from "./tables" |
|||
import { fetchViewData } from "./views" |
|||
import { fetchRelationshipData } from "./relationships" |
|||
import { FieldTypes } from "../constants" |
|||
import { executeQuery, fetchQueryDefinition } from "./queries" |
|||
import { |
|||
convertJSONSchemaToTableSchema, |
|||
getJSONArrayDatasourceSchema, |
|||
} from "builder/src/builderStore/jsonUtils" |
|||
|
|||
/** |
|||
* Fetches all rows for a particular Budibase data source. |
|||
*/ |
|||
export const fetchDatasource = async dataSource => { |
|||
if (!dataSource || !dataSource.type) { |
|||
return [] |
|||
} |
|||
|
|||
// Fetch all rows in data source
|
|||
const { type, tableId, fieldName } = dataSource |
|||
let rows = [], |
|||
info = {} |
|||
if (type === "table") { |
|||
rows = await fetchTableData(tableId) |
|||
} else if (type === "view") { |
|||
rows = await fetchViewData(dataSource) |
|||
} else if (type === "query") { |
|||
// Set the default query params
|
|||
let parameters = cloneDeep(dataSource.queryParams || {}) |
|||
for (let param of dataSource.parameters) { |
|||
if (!parameters[param.name]) { |
|||
parameters[param.name] = param.default |
|||
} |
|||
} |
|||
const { data, ...rest } = await executeQuery({ |
|||
queryId: dataSource._id, |
|||
parameters, |
|||
}) |
|||
info = rest |
|||
rows = data |
|||
} else if (type === FieldTypes.LINK) { |
|||
rows = await fetchRelationshipData({ |
|||
rowId: dataSource.rowId, |
|||
tableId: dataSource.rowTableId, |
|||
fieldName, |
|||
}) |
|||
} |
|||
|
|||
// Enrich the result is always an array
|
|||
return { rows: Array.isArray(rows) ? rows : [], info } |
|||
} |
|||
|
|||
/** |
|||
* Fetches the schema of any kind of datasource. |
|||
*/ |
|||
export const fetchDatasourceSchema = async dataSource => { |
|||
if (!dataSource) { |
|||
return null |
|||
} |
|||
const { type } = dataSource |
|||
let schema |
|||
|
|||
// Nested providers should already have exposed their own schema
|
|||
if (type === "provider") { |
|||
schema = dataSource.value?.schema |
|||
} |
|||
|
|||
// Field sources have their schema statically defined
|
|||
if (type === "field") { |
|||
if (dataSource.fieldType === "attachment") { |
|||
schema = { |
|||
url: { |
|||
type: "string", |
|||
}, |
|||
name: { |
|||
type: "string", |
|||
}, |
|||
} |
|||
} else if (dataSource.fieldType === "array") { |
|||
schema = { |
|||
value: { |
|||
type: "string", |
|||
}, |
|||
} |
|||
} |
|||
} |
|||
|
|||
// JSON arrays need their table definitions fetched.
|
|||
// We can then extract their schema as a subset of the table schema.
|
|||
if (type === "jsonarray") { |
|||
const table = await fetchTableDefinition(dataSource.tableId) |
|||
schema = getJSONArrayDatasourceSchema(table?.schema, dataSource) |
|||
} |
|||
|
|||
// Tables, views and links can be fetched by table ID
|
|||
if ( |
|||
(type === "table" || type === "view" || type === "link") && |
|||
dataSource.tableId |
|||
) { |
|||
const table = await fetchTableDefinition(dataSource.tableId) |
|||
schema = table?.schema |
|||
} |
|||
|
|||
// Queries can be fetched by query ID
|
|||
if (type === "query" && dataSource._id) { |
|||
const definition = await fetchQueryDefinition(dataSource._id) |
|||
schema = definition?.schema |
|||
} |
|||
|
|||
// Sanity check
|
|||
if (!schema) { |
|||
return null |
|||
} |
|||
|
|||
// Check for any JSON fields so we can add any top level properties
|
|||
let jsonAdditions = {} |
|||
Object.keys(schema).forEach(fieldKey => { |
|||
const fieldSchema = schema[fieldKey] |
|||
if (fieldSchema?.type === "json") { |
|||
const jsonSchema = convertJSONSchemaToTableSchema(fieldSchema, { |
|||
squashObjects: true, |
|||
}) |
|||
Object.keys(jsonSchema).forEach(jsonKey => { |
|||
jsonAdditions[`${fieldKey}.${jsonKey}`] = { |
|||
type: jsonSchema[jsonKey].type, |
|||
nestedJSON: true, |
|||
} |
|||
}) |
|||
} |
|||
}) |
|||
return { ...schema, ...jsonAdditions } |
|||
} |
|||
@ -0,0 +1,407 @@ |
|||
import { writable, derived, get } from "svelte/store" |
|||
import { |
|||
buildLuceneQuery, |
|||
luceneLimit, |
|||
luceneQuery, |
|||
luceneSort, |
|||
} from "builder/src/helpers/lucene" |
|||
import { fetchTableDefinition } from "api" |
|||
|
|||
/** |
|||
* Parent class which handles the implementation of fetching data from an |
|||
* internal table or datasource plus. |
|||
* For other types of datasource, this class is overridden and extended. |
|||
*/ |
|||
export default class DataFetch { |
|||
// Feature flags
|
|||
featureStore = writable({ |
|||
supportsSearch: false, |
|||
supportsSort: false, |
|||
supportsPagination: false, |
|||
}) |
|||
|
|||
// Config
|
|||
options = { |
|||
datasource: null, |
|||
limit: 10, |
|||
|
|||
// Search config
|
|||
filter: null, |
|||
query: null, |
|||
|
|||
// Sorting config
|
|||
sortColumn: null, |
|||
sortOrder: "ascending", |
|||
sortType: null, |
|||
|
|||
// Pagination config
|
|||
paginate: true, |
|||
} |
|||
|
|||
// State of the fetch
|
|||
store = writable({ |
|||
rows: [], |
|||
info: null, |
|||
schema: null, |
|||
loading: false, |
|||
loaded: false, |
|||
query: null, |
|||
pageNumber: 0, |
|||
cursor: null, |
|||
cursors: [], |
|||
}) |
|||
|
|||
/** |
|||
* Constructs a new DataFetch instance. |
|||
* @param opts the fetch options |
|||
*/ |
|||
constructor(opts) { |
|||
// Merge options with their default values
|
|||
this.options = { |
|||
...this.options, |
|||
...opts, |
|||
} |
|||
|
|||
// Bind all functions to properly scope "this"
|
|||
this.getData = this.getData.bind(this) |
|||
this.getPage = this.getPage.bind(this) |
|||
this.getInitialData = this.getInitialData.bind(this) |
|||
this.determineFeatureFlags = this.determineFeatureFlags.bind(this) |
|||
this.enrichSchema = this.enrichSchema.bind(this) |
|||
this.refresh = this.refresh.bind(this) |
|||
this.update = this.update.bind(this) |
|||
this.hasNextPage = this.hasNextPage.bind(this) |
|||
this.hasPrevPage = this.hasPrevPage.bind(this) |
|||
this.nextPage = this.nextPage.bind(this) |
|||
this.prevPage = this.prevPage.bind(this) |
|||
|
|||
// Derive certain properties to return
|
|||
this.derivedStore = derived( |
|||
[this.store, this.featureStore], |
|||
([$store, $featureStore]) => { |
|||
return { |
|||
...$store, |
|||
...$featureStore, |
|||
hasNextPage: this.hasNextPage($store), |
|||
hasPrevPage: this.hasPrevPage($store), |
|||
} |
|||
} |
|||
) |
|||
|
|||
// Mark as loaded if we have no datasource
|
|||
if (!this.options.datasource) { |
|||
this.store.update($store => ({ ...$store, loaded: true })) |
|||
return |
|||
} |
|||
|
|||
// Initially fetch data but don't bother waiting for the result
|
|||
this.getInitialData() |
|||
} |
|||
|
|||
/** |
|||
* Extend the svelte store subscribe method to that instances of this class |
|||
* can be treated like stores |
|||
*/ |
|||
get subscribe() { |
|||
return this.derivedStore.subscribe |
|||
} |
|||
|
|||
/** |
|||
* Fetches a fresh set of data from the server, resetting pagination |
|||
*/ |
|||
async getInitialData() { |
|||
const { datasource, filter, sortColumn, paginate } = this.options |
|||
const tableId = datasource?.tableId |
|||
|
|||
// Ensure table ID exists
|
|||
if (!tableId) { |
|||
return |
|||
} |
|||
|
|||
// Fetch datasource definition and determine feature flags
|
|||
const definition = await this.constructor.getDefinition(datasource) |
|||
const features = this.determineFeatureFlags(definition) |
|||
this.featureStore.set({ |
|||
supportsSearch: !!features?.supportsSearch, |
|||
supportsSort: !!features?.supportsSort, |
|||
supportsPagination: paginate && !!features?.supportsPagination, |
|||
}) |
|||
|
|||
// Fetch and enrich schema
|
|||
let schema = this.constructor.getSchema(datasource, definition) |
|||
schema = this.enrichSchema(schema) |
|||
if (!schema) { |
|||
return |
|||
} |
|||
|
|||
// Determine what sort type to use
|
|||
if (!this.options.sortType) { |
|||
let sortType = "string" |
|||
if (sortColumn) { |
|||
const type = schema?.[sortColumn]?.type |
|||
sortType = type === "number" ? "number" : "string" |
|||
} |
|||
this.options.sortType = sortType |
|||
} |
|||
|
|||
// Build the lucene query
|
|||
let query = this.options.query |
|||
if (!query) { |
|||
query = buildLuceneQuery(filter) |
|||
} |
|||
|
|||
// Update store
|
|||
this.store.update($store => ({ |
|||
...$store, |
|||
definition, |
|||
schema, |
|||
query, |
|||
loading: true, |
|||
})) |
|||
|
|||
// Actually fetch data
|
|||
const page = await this.getPage() |
|||
this.store.update($store => ({ |
|||
...$store, |
|||
loading: false, |
|||
loaded: true, |
|||
pageNumber: 0, |
|||
rows: page.rows, |
|||
info: page.info, |
|||
cursors: paginate && page.hasNextPage ? [null, page.cursor] : [null], |
|||
})) |
|||
} |
|||
|
|||
/** |
|||
* Fetches some filtered, sorted and paginated data |
|||
*/ |
|||
async getPage() { |
|||
const { sortColumn, sortOrder, sortType, limit } = this.options |
|||
const { query } = get(this.store) |
|||
const features = get(this.featureStore) |
|||
|
|||
// Get the actual data
|
|||
let { rows, info, hasNextPage, cursor } = await this.getData() |
|||
|
|||
// If we don't support searching, do a client search
|
|||
if (!features.supportsSearch) { |
|||
rows = luceneQuery(rows, query) |
|||
} |
|||
|
|||
// If we don't support sorting, do a client-side sort
|
|||
if (!features.supportsSort) { |
|||
rows = luceneSort(rows, sortColumn, sortOrder, sortType) |
|||
} |
|||
|
|||
// If we don't support pagination, do a client-side limit
|
|||
if (!features.supportsPagination) { |
|||
rows = luceneLimit(rows, limit) |
|||
} |
|||
|
|||
return { |
|||
rows, |
|||
info, |
|||
hasNextPage, |
|||
cursor, |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* Fetches a single page of data from the remote resource. |
|||
* Must be overridden by a datasource specific child class. |
|||
*/ |
|||
async getData() { |
|||
return { |
|||
rows: [], |
|||
info: null, |
|||
hasNextPage: false, |
|||
cursor: null, |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* Gets the definition for this datasource. |
|||
* Defaults to fetching a table definition. |
|||
* @param datasource |
|||
* @return {object} the definition |
|||
*/ |
|||
static async getDefinition(datasource) { |
|||
if (!datasource?.tableId) { |
|||
return null |
|||
} |
|||
return await fetchTableDefinition(datasource.tableId) |
|||
} |
|||
|
|||
/** |
|||
* Gets the schema definition for a datasource. |
|||
* Defaults to getting the "schema" property of the definition. |
|||
* @param datasource the datasource |
|||
* @param definition the datasource definition |
|||
* @return {object} the schema |
|||
*/ |
|||
static getSchema(datasource, definition) { |
|||
return definition?.schema |
|||
} |
|||
|
|||
/** |
|||
* Enriches the schema and ensures that entries are objects with names |
|||
* @param schema the datasource schema |
|||
* @return {object} the enriched datasource schema |
|||
*/ |
|||
enrichSchema(schema) { |
|||
if (schema == null) { |
|||
return null |
|||
} |
|||
let enrichedSchema = {} |
|||
Object.entries(schema).forEach(([fieldName, fieldSchema]) => { |
|||
if (typeof fieldSchema === "string") { |
|||
enrichedSchema[fieldName] = { |
|||
type: fieldSchema, |
|||
name: fieldName, |
|||
} |
|||
} else { |
|||
enrichedSchema[fieldName] = { |
|||
...fieldSchema, |
|||
name: fieldName, |
|||
} |
|||
} |
|||
}) |
|||
return enrichedSchema |
|||
} |
|||
|
|||
/** |
|||
* Determine the feature flag for this datasource definition |
|||
* @param definition |
|||
*/ |
|||
// eslint-disable-next-line no-unused-vars
|
|||
determineFeatureFlags(definition) { |
|||
return { |
|||
supportsSearch: false, |
|||
supportsSort: false, |
|||
supportsPagination: false, |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* Resets the data set and updates options |
|||
* @param newOptions any new options |
|||
*/ |
|||
async update(newOptions) { |
|||
// Check if any settings have actually changed
|
|||
let refresh = false |
|||
const entries = Object.entries(newOptions || {}) |
|||
for (let [key, value] of entries) { |
|||
if (JSON.stringify(value) !== JSON.stringify(this.options[key])) { |
|||
refresh = true |
|||
break |
|||
} |
|||
} |
|||
if (!refresh) { |
|||
return |
|||
} |
|||
|
|||
// Assign new options and reload data
|
|||
this.options = { |
|||
...this.options, |
|||
...newOptions, |
|||
} |
|||
await this.getInitialData() |
|||
} |
|||
|
|||
/** |
|||
* Loads the same page again |
|||
*/ |
|||
async refresh() { |
|||
if (get(this.store).loading) { |
|||
return |
|||
} |
|||
this.store.update($store => ({ ...$store, loading: true })) |
|||
const { rows, info } = await this.getPage() |
|||
this.store.update($store => ({ ...$store, rows, info, loading: false })) |
|||
} |
|||
|
|||
/** |
|||
* Determines whether there is a next page of data based on the state of the |
|||
* store |
|||
* @param state the current store state |
|||
* @return {boolean} whether there is a next page of data or not |
|||
*/ |
|||
hasNextPage(state) { |
|||
return state.cursors[state.pageNumber + 1] != null |
|||
} |
|||
|
|||
/** |
|||
* Determines whether there is a previous page of data based on the state of |
|||
* the store |
|||
* @param state the current store state |
|||
* @return {boolean} whether there is a previous page of data or not |
|||
*/ |
|||
hasPrevPage(state) { |
|||
return state.pageNumber > 0 |
|||
} |
|||
|
|||
/** |
|||
* Fetches the next page of data |
|||
*/ |
|||
async nextPage() { |
|||
const state = get(this.derivedStore) |
|||
if (state.loading || !this.options.paginate || !state.hasNextPage) { |
|||
return |
|||
} |
|||
|
|||
// Fetch next page
|
|||
const nextCursor = state.cursors[state.pageNumber + 1] |
|||
this.store.update($store => ({ |
|||
...$store, |
|||
loading: true, |
|||
cursor: nextCursor, |
|||
pageNumber: $store.pageNumber + 1, |
|||
})) |
|||
const { rows, info, hasNextPage, cursor } = await this.getPage() |
|||
|
|||
// Update state
|
|||
this.store.update($store => { |
|||
let { cursors, pageNumber } = $store |
|||
if (hasNextPage) { |
|||
cursors[pageNumber + 1] = cursor |
|||
} |
|||
return { |
|||
...$store, |
|||
rows, |
|||
info, |
|||
cursors, |
|||
loading: false, |
|||
} |
|||
}) |
|||
} |
|||
|
|||
/** |
|||
* Fetches the previous page of data |
|||
*/ |
|||
async prevPage() { |
|||
const state = get(this.derivedStore) |
|||
if (state.loading || !this.options.paginate || !state.hasPrevPage) { |
|||
return |
|||
} |
|||
|
|||
// Fetch previous page
|
|||
const prevCursor = state.cursors[state.pageNumber - 1] |
|||
this.store.update($store => ({ |
|||
...$store, |
|||
loading: true, |
|||
cursor: prevCursor, |
|||
pageNumber: $store.pageNumber - 1, |
|||
})) |
|||
const { rows, info } = await this.getPage() |
|||
|
|||
// Update state
|
|||
this.store.update($store => { |
|||
return { |
|||
...$store, |
|||
rows, |
|||
info, |
|||
loading: false, |
|||
} |
|||
}) |
|||
} |
|||
} |
|||
@ -0,0 +1,44 @@ |
|||
import DataFetch from "./DataFetch.js" |
|||
|
|||
export default class FieldFetch extends DataFetch { |
|||
static async getDefinition(datasource) { |
|||
// Field sources have their schema statically defined
|
|||
let schema |
|||
if (datasource.fieldType === "attachment") { |
|||
schema = { |
|||
url: { |
|||
type: "string", |
|||
}, |
|||
name: { |
|||
type: "string", |
|||
}, |
|||
} |
|||
} else if (datasource.fieldType === "array") { |
|||
schema = { |
|||
value: { |
|||
type: "string", |
|||
}, |
|||
} |
|||
} |
|||
return { schema } |
|||
} |
|||
|
|||
async getData() { |
|||
const { datasource } = this.options |
|||
|
|||
// These sources will be available directly from context
|
|||
const data = datasource?.value || [] |
|||
let rows = [] |
|||
if (Array.isArray(data) && data[0] && typeof data[0] !== "object") { |
|||
rows = data.map(value => ({ value })) |
|||
} else { |
|||
rows = data |
|||
} |
|||
|
|||
return { |
|||
rows: rows || [], |
|||
hasNextPage: false, |
|||
cursor: null, |
|||
} |
|||
} |
|||
} |
|||
@ -0,0 +1,13 @@ |
|||
import FieldFetch from "./FieldFetch.js" |
|||
import { fetchTableDefinition } from "api" |
|||
import { getJSONArrayDatasourceSchema } from "builder/src/builderStore/jsonUtils" |
|||
|
|||
export default class JSONArrayFetch extends FieldFetch { |
|||
static async getDefinition(datasource) { |
|||
// JSON arrays need their table definitions fetched.
|
|||
// We can then extract their schema as a subset of the table schema.
|
|||
const table = await fetchTableDefinition(datasource.tableId) |
|||
const schema = getJSONArrayDatasourceSchema(table?.schema, datasource) |
|||
return { schema } |
|||
} |
|||
} |
|||
@ -0,0 +1,20 @@ |
|||
import DataFetch from "./DataFetch.js" |
|||
|
|||
export default class NestedProviderFetch extends DataFetch { |
|||
static async getDefinition(datasource) { |
|||
// Nested providers should already have exposed their own schema
|
|||
return { |
|||
schema: datasource?.value?.schema, |
|||
} |
|||
} |
|||
|
|||
async getData() { |
|||
const { datasource } = this.options |
|||
// Pull the rows from the existing data provider
|
|||
return { |
|||
rows: datasource?.value?.rows || [], |
|||
hasNextPage: false, |
|||
cursor: null, |
|||
} |
|||
} |
|||
} |
|||
@ -0,0 +1,68 @@ |
|||
import DataFetch from "./DataFetch.js" |
|||
import { executeQuery, fetchQueryDefinition } from "api" |
|||
import { cloneDeep } from "lodash/fp" |
|||
import { get } from "svelte/store" |
|||
|
|||
export default class QueryFetch extends DataFetch { |
|||
determineFeatureFlags(definition) { |
|||
const supportsPagination = |
|||
!!definition?.fields?.pagination?.type && |
|||
!!definition?.fields?.pagination?.location && |
|||
!!definition?.fields?.pagination?.pageParam |
|||
return { supportsPagination } |
|||
} |
|||
|
|||
static async getDefinition(datasource) { |
|||
if (!datasource?._id) { |
|||
return null |
|||
} |
|||
return await fetchQueryDefinition(datasource._id) |
|||
} |
|||
|
|||
async getData() { |
|||
const { datasource, limit, paginate } = this.options |
|||
const { supportsPagination } = get(this.featureStore) |
|||
const { cursor, definition } = get(this.store) |
|||
const type = definition?.fields?.pagination?.type |
|||
|
|||
// Set the default query params
|
|||
let parameters = cloneDeep(datasource?.queryParams || {}) |
|||
for (let param of datasource?.parameters || {}) { |
|||
if (!parameters[param.name]) { |
|||
parameters[param.name] = param.default |
|||
} |
|||
} |
|||
|
|||
// Add pagination to query if supported
|
|||
let queryPayload = { queryId: datasource?._id, parameters } |
|||
if (paginate && supportsPagination) { |
|||
const requestCursor = type === "page" ? parseInt(cursor || 1) : cursor |
|||
queryPayload.pagination = { page: requestCursor, limit } |
|||
} |
|||
|
|||
// Execute query
|
|||
const { data, pagination, ...rest } = await executeQuery(queryPayload) |
|||
|
|||
// Derive pagination info from response
|
|||
let nextCursor = null |
|||
let hasNextPage = false |
|||
if (paginate && supportsPagination) { |
|||
if (type === "page") { |
|||
// For "page number" pagination, increment the existing page number
|
|||
nextCursor = queryPayload.pagination.page + 1 |
|||
hasNextPage = data?.length === limit && limit > 0 |
|||
} else { |
|||
// For "cursor" pagination, the cursor should be in the response
|
|||
nextCursor = pagination?.cursor |
|||
hasNextPage = nextCursor != null |
|||
} |
|||
} |
|||
|
|||
return { |
|||
rows: data || [], |
|||
info: rest, |
|||
cursor: nextCursor, |
|||
hasNextPage, |
|||
} |
|||
} |
|||
} |
|||
@ -0,0 +1,16 @@ |
|||
import DataFetch from "./DataFetch.js" |
|||
import { fetchRelationshipData } from "api" |
|||
|
|||
export default class RelationshipFetch extends DataFetch { |
|||
async getData() { |
|||
const { datasource } = this.options |
|||
const res = await fetchRelationshipData({ |
|||
rowId: datasource?.rowId, |
|||
tableId: datasource?.rowTableId, |
|||
fieldName: datasource?.fieldName, |
|||
}) |
|||
return { |
|||
rows: res || [], |
|||
} |
|||
} |
|||
} |
|||
@ -0,0 +1,37 @@ |
|||
import { get } from "svelte/store" |
|||
import DataFetch from "./DataFetch.js" |
|||
import { searchTable } from "api" |
|||
|
|||
export default class TableFetch extends DataFetch { |
|||
determineFeatureFlags() { |
|||
return { |
|||
supportsSearch: true, |
|||
supportsSort: true, |
|||
supportsPagination: true, |
|||
} |
|||
} |
|||
|
|||
async getData() { |
|||
const { datasource, limit, sortColumn, sortOrder, sortType, paginate } = |
|||
this.options |
|||
const { tableId } = datasource |
|||
const { cursor, query } = get(this.store) |
|||
|
|||
// Search table
|
|||
const res = await searchTable({ |
|||
tableId, |
|||
query, |
|||
limit, |
|||
sort: sortColumn, |
|||
sortOrder: sortOrder?.toLowerCase() ?? "ascending", |
|||
sortType, |
|||
paginate, |
|||
bookmark: cursor, |
|||
}) |
|||
return { |
|||
rows: res?.rows || [], |
|||
hasNextPage: res?.hasNextPage || false, |
|||
cursor: res?.bookmark || null, |
|||
} |
|||
} |
|||
} |
|||
@ -0,0 +1,16 @@ |
|||
import DataFetch from "./DataFetch.js" |
|||
import { fetchViewData } from "api" |
|||
|
|||
export default class ViewFetch extends DataFetch { |
|||
static getSchema(datasource, definition) { |
|||
return definition?.views?.[datasource.name]?.schema |
|||
} |
|||
|
|||
async getData() { |
|||
const { datasource } = this.options |
|||
const res = await fetchViewData(datasource) |
|||
return { |
|||
rows: res || [], |
|||
} |
|||
} |
|||
} |
|||
@ -0,0 +1,22 @@ |
|||
import TableFetch from "./TableFetch.js" |
|||
import ViewFetch from "./ViewFetch.js" |
|||
import QueryFetch from "./QueryFetch.js" |
|||
import RelationshipFetch from "./RelationshipFetch.js" |
|||
import NestedProviderFetch from "./NestedProviderFetch.js" |
|||
import FieldFetch from "./FieldFetch.js" |
|||
import JSONArrayFetch from "./JSONArrayFetch.js" |
|||
|
|||
const DataFetchMap = { |
|||
table: TableFetch, |
|||
view: ViewFetch, |
|||
query: QueryFetch, |
|||
link: RelationshipFetch, |
|||
provider: NestedProviderFetch, |
|||
field: FieldFetch, |
|||
jsonarray: JSONArrayFetch, |
|||
} |
|||
|
|||
export const fetchData = (datasource, options) => { |
|||
const Fetch = DataFetchMap[datasource?.type] || TableFetch |
|||
return new Fetch({ datasource, ...options }) |
|||
} |
|||
@ -0,0 +1,53 @@ |
|||
import { convertJSONSchemaToTableSchema } from "builder/src/builderStore/jsonUtils" |
|||
import TableFetch from "./fetch/TableFetch.js" |
|||
import ViewFetch from "./fetch/ViewFetch.js" |
|||
import QueryFetch from "./fetch/QueryFetch.js" |
|||
import RelationshipFetch from "./fetch/RelationshipFetch.js" |
|||
import NestedProviderFetch from "./fetch/NestedProviderFetch.js" |
|||
import FieldFetch from "./fetch/FieldFetch.js" |
|||
import JSONArrayFetch from "./fetch/JSONArrayFetch.js" |
|||
|
|||
/** |
|||
* Fetches the schema of any kind of datasource. |
|||
* All datasource fetch classes implement their own functionality to get the |
|||
* schema of a datasource of their respective types. |
|||
*/ |
|||
export const fetchDatasourceSchema = async datasource => { |
|||
const handler = { |
|||
table: TableFetch, |
|||
view: ViewFetch, |
|||
query: QueryFetch, |
|||
link: RelationshipFetch, |
|||
provider: NestedProviderFetch, |
|||
field: FieldFetch, |
|||
jsonarray: JSONArrayFetch, |
|||
}[datasource?.type] |
|||
if (!handler) { |
|||
return null |
|||
} |
|||
|
|||
// Get the datasource definition and then schema
|
|||
const definition = await handler.getDefinition(datasource) |
|||
const schema = handler.getSchema(datasource, definition) |
|||
if (!schema) { |
|||
return null |
|||
} |
|||
|
|||
// Check for any JSON fields so we can add any top level properties
|
|||
let jsonAdditions = {} |
|||
Object.keys(schema).forEach(fieldKey => { |
|||
const fieldSchema = schema[fieldKey] |
|||
if (fieldSchema?.type === "json") { |
|||
const jsonSchema = convertJSONSchemaToTableSchema(fieldSchema, { |
|||
squashObjects: true, |
|||
}) |
|||
Object.keys(jsonSchema).forEach(jsonKey => { |
|||
jsonAdditions[`${fieldKey}.${jsonKey}`] = { |
|||
type: jsonSchema[jsonKey].type, |
|||
nestedJSON: true, |
|||
} |
|||
}) |
|||
} |
|||
}) |
|||
return { ...schema, ...jsonAdditions } |
|||
} |
|||
@ -0,0 +1,161 @@ |
|||
const { processString } = require("@budibase/string-templates") |
|||
const CouchDB = require("../../db") |
|||
const { |
|||
generateQueryID, |
|||
getQueryParams, |
|||
isProdAppID, |
|||
} = require("../../db/utils") |
|||
const { BaseQueryVerbs } = require("../../constants") |
|||
const { Thread, ThreadType } = require("../../threads") |
|||
const env = require("../../environment") |
|||
|
|||
const Runner = new Thread(ThreadType.QUERY, { |
|||
timeoutMs: env.QUERY_THREAD_TIMEOUT || 10000, |
|||
}) |
|||
|
|||
// simple function to append "readable" to all read queries
|
|||
function enrichQueries(input) { |
|||
const wasArray = Array.isArray(input) |
|||
const queries = wasArray ? input : [input] |
|||
for (let query of queries) { |
|||
if (query.queryVerb === BaseQueryVerbs.READ) { |
|||
query.readable = true |
|||
} |
|||
} |
|||
return wasArray ? queries : queries[0] |
|||
} |
|||
|
|||
exports.fetch = async function (ctx) { |
|||
const db = new CouchDB(ctx.appId) |
|||
|
|||
const body = await db.allDocs( |
|||
getQueryParams(null, { |
|||
include_docs: true, |
|||
}) |
|||
) |
|||
ctx.body = enrichQueries(body.rows.map(row => row.doc)) |
|||
} |
|||
|
|||
exports.save = async function (ctx) { |
|||
const db = new CouchDB(ctx.appId) |
|||
const query = ctx.request.body |
|||
|
|||
if (!query._id) { |
|||
query._id = generateQueryID(query.datasourceId) |
|||
} |
|||
|
|||
const response = await db.put(query) |
|||
query._rev = response.rev |
|||
|
|||
ctx.body = query |
|||
ctx.message = `Query ${query.name} saved successfully.` |
|||
} |
|||
|
|||
async function enrichQueryFields(fields, parameters = {}) { |
|||
const enrichedQuery = {} |
|||
|
|||
// enrich the fields with dynamic parameters
|
|||
for (let key of Object.keys(fields)) { |
|||
if (fields[key] == null) { |
|||
continue |
|||
} |
|||
if (typeof fields[key] === "object") { |
|||
// enrich nested fields object
|
|||
enrichedQuery[key] = await enrichQueryFields(fields[key], parameters) |
|||
} else if (typeof fields[key] === "string") { |
|||
// enrich string value as normal
|
|||
enrichedQuery[key] = await processString(fields[key], parameters, { |
|||
noHelpers: true, |
|||
}) |
|||
} else { |
|||
enrichedQuery[key] = fields[key] |
|||
} |
|||
} |
|||
|
|||
if ( |
|||
enrichedQuery.json || |
|||
enrichedQuery.customData || |
|||
enrichedQuery.requestBody |
|||
) { |
|||
try { |
|||
enrichedQuery.json = JSON.parse( |
|||
enrichedQuery.json || |
|||
enrichedQuery.customData || |
|||
enrichedQuery.requestBody |
|||
) |
|||
} catch (err) { |
|||
throw { message: `JSON Invalid - error: ${err}` } |
|||
} |
|||
delete enrichedQuery.customData |
|||
} |
|||
|
|||
return enrichedQuery |
|||
} |
|||
|
|||
exports.find = async function (ctx) { |
|||
const db = new CouchDB(ctx.appId) |
|||
const query = enrichQueries(await db.get(ctx.params.queryId)) |
|||
// remove properties that could be dangerous in real app
|
|||
if (isProdAppID(ctx.appId)) { |
|||
delete query.fields |
|||
delete query.parameters |
|||
} |
|||
ctx.body = query |
|||
} |
|||
|
|||
exports.preview = async function (ctx) { |
|||
const db = new CouchDB(ctx.appId) |
|||
|
|||
const datasource = await db.get(ctx.request.body.datasourceId) |
|||
|
|||
const { fields, parameters, queryVerb, transformer } = ctx.request.body |
|||
const enrichedQuery = await enrichQueryFields(fields, parameters) |
|||
|
|||
try { |
|||
const { rows, keys } = await Runner.run({ |
|||
datasource, |
|||
queryVerb, |
|||
query: enrichedQuery, |
|||
transformer, |
|||
}) |
|||
|
|||
ctx.body = { |
|||
rows, |
|||
schemaFields: [...new Set(keys)], |
|||
} |
|||
} catch (err) { |
|||
ctx.throw(400, err) |
|||
} |
|||
} |
|||
|
|||
exports.execute = async function (ctx) { |
|||
const db = new CouchDB(ctx.appId) |
|||
|
|||
const query = await db.get(ctx.params.queryId) |
|||
const datasource = await db.get(query.datasourceId) |
|||
|
|||
const enrichedQuery = await enrichQueryFields( |
|||
query.fields, |
|||
ctx.request.body.parameters |
|||
) |
|||
|
|||
// call the relevant CRUD method on the integration class
|
|||
try { |
|||
const { rows } = await Runner.run({ |
|||
datasource, |
|||
queryVerb: query.queryVerb, |
|||
query: enrichedQuery, |
|||
transformer: query.transformer, |
|||
}) |
|||
ctx.body = rows |
|||
} catch (err) { |
|||
ctx.throw(400, err) |
|||
} |
|||
} |
|||
|
|||
exports.destroy = async function (ctx) { |
|||
const db = new CouchDB(ctx.appId) |
|||
await db.remove(ctx.params.queryId, ctx.params.revId) |
|||
ctx.message = `Query deleted.` |
|||
ctx.status = 200 |
|||
} |
|||
Loading…
Reference in new issue