diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 7abfe537e..d1e373003 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -79,6 +79,8 @@ Component libraries are collections of components as well as the definition of t ### Getting Started For Contributors #### 1. Prerequisites +NodeJS Version `14.x.x` + *yarn -* `npm install -g yarn` *jest* - `npm install -g jest` @@ -177,36 +179,7 @@ To enable this mode, use: yarn mode:account ``` ### CI - -#### PR Job - -After your pr is submitted a github action (can be found at `.github/workflows/budibase_ci.yml`) will run to perform some checks against the changes such as linting, build and test. - -The job will run when changes are pushed to or targetted at `master` and `develop` -#### Release Develop - -To test changes before a release, a prerelease action (can be found at `.github/workflows/release-develop.yml`) will run to build and release develop versions of npm packages and docker images. On each subsequent commit to develop a new alpha version of npm packages will be created and released. - -For example: - -- `feature1` -> `develop` = `v0.9.160-alpha.1` -- `feature2` -> `develop` = `v0.9.160-alpha.0` - -The job will run when changes are pushed to `develop` -#### Release Job - -To release changes a release job (can be found at `.github/workflows/release.yml`) will run to create final versions of npm packages and docker images. - -Following the example above: - -- `develop` -> `master` = `v0.9.160` - -The job will run when changes are pushed to `master` - -#### Release Self Host Job - -To release the self hosted version of docker images, an additional job (can be found at `.github/workflows/release-selfhost.yml`) must be ran manually. This will releaae docker images to docker hub under the tag `latest` to be picked up by self hosted installations. - + An overview of the CI pipelines can be found [here](./workflows/README.md) ### Troubleshooting Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above. You should have a fresh Budibase installation. diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 000000000..d2fcd16bb --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,93 @@ + +# Budibase CI Pipelines + +Welcome to the budibase CI pipelines directory. This document details what each of the CI pipelines are for, and come common combinations. + +## All CI Pipelines + +### Note +- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. + +### Standard CI Build Job (budibase_ci.yml) +Triggers: +- PR or push to develop +- PR or push to master + +The standard CI Build job is what runs when you raise a PR to develop or master. +- Installs all dependencies, +- builds the project +- run the unit tests +- Generate test coverage metrics with codecov +- Run the cypress tests + +### Release Develop Job (release-develop.yml) +Triggers: +- Push to develop + +The job responsible for building, tagging and pushing docker images out to the test and staging environments. +- Installs all dependencies +- builds the project +- run the unit tests +- publish the budibase JS packages under a prerelease tag to NPM +- build, tag and push docker images under the `develop` tag to docker hub + +These images will then be pulled by the test and staging environments, updating the latest automatically. Discord notifications are sent to the #infra channel when this occurs. + +### Release Job (release.yml) +Triggers: +- Push to master + +This job is responsible for building and pushing the latest code to NPM and docker hub, so that it can be deployed. +- Installs all dependencies +- builds the project +- run the unit tests +- publish the budibase JS packages under a release tag to NPM (always incremented by patch versions) +- build, tag and push docker images under the `v.x.x.x` (the tag of the NPM release) tag to docker hub + +### Release Selfhost Job (release-selfhost.yml) +Triggers: +- Manual Workflow Dispatch Trigger + +This job is responsible for delivering the latest version of budibase to those that are self-hosting. + +This job relies on the release job to have run first, so the latest image is pushed to dockerhub. This job then will pull the latest version from `lerna.json` and try to find an image in dockerhub corresponding to that version. For example, if the version in `lerna.json` is `1.0.0`: +- Pull the images for all budibase services tagged `v1.0.0` from dockerhub +- Tag these images as `latest` +- Push them back to dockerhub. This now means anyone who pulls `latest` (self hosters using docker-compose) will get the latest version. +- Build and release the budibase helm chart for kubernetes users +- Perform a github release with the latest version. You can see previous releases here (https://github.com/Budibase/budibase/releases) + + +### Cloud Deploy (deploy-cloud.yml) +Triggers: +- Manual Workflow Dispatch Trigger + +This job is responsible for deploying to our production, cloud kubernetes environment. You must run the release job first, to ensure that the latest images have been built and pushed to docker hub. You can also manually enter a version number for this job, so you can perform rollbacks or upgrade to a specific version. After kicking off this job, the following will occur: + +- Checks out the master branch +- Pulls the latest `values.yaml` from budibase infra, a private repo containing budibases infrastructure configuration +- Gets the latest budibase version from `lerna.json`, if it hasn't been specified in the workflow when you kicked it off +- Configures AWS Credentials +- Deploys the helm chart in the budibase repo to our production EKS cluster, injecting the `values.yaml` we pulled from budibase-infra +- Fires off a discord webhook in the #infra channel to show that the deployment completely successfully. + +## Common Workflows + +### Deploy Changes to Production (Release) +- Merge `develop` into `master` +- Wait for budibase CI job and release job to run +- Run cloud deploy job +- Run release selfhost job + +### Deploy Changes to Production (Hotfix) +- Branch off `master` +- Perform your hotfix +- Merge back into `master` +- Wait for budibase CI job and release job to run +- Run cloud deploy job +- Run release selfhost job + +### Rollback A Bad Cloud Deployment +- Kick off cloud deploy job +- Ensure you are running off master +- Enter the version number of the last known good version of budibase. For example `1.0.0` \ No newline at end of file diff --git a/.github/workflows/budibase_ci.yml b/.github/workflows/budibase_ci.yml index 5c4a111e2..7e9511541 100644 --- a/.github/workflows/budibase_ci.yml +++ b/.github/workflows/budibase_ci.yml @@ -41,4 +41,6 @@ jobs: files: ./packages/server/coverage/clover.xml name: codecov-umbrella verbose: true + + # TODO: parallelise this - run: yarn test:e2e:ci diff --git a/.github/workflows/deploy-cloud.yaml b/.github/workflows/deploy-cloud.yaml index 26422a2e7..d54e6c9c6 100644 --- a/.github/workflows/deploy-cloud.yaml +++ b/.github/workflows/deploy-cloud.yaml @@ -12,6 +12,12 @@ jobs: runs-on: ubuntu-latest steps: + - name: Fail if branch is not master + if: github.ref != 'refs/heads/master' + run: | + echo "Ref is not master, you must run this job from master." + exit 1 + - uses: actions/checkout@v2 - name: Pull values.yaml from budibase-infra diff --git a/.github/workflows/deploy-preprod.yml b/.github/workflows/deploy-preprod.yml index ce789e71e..5b3282313 100644 --- a/.github/workflows/deploy-preprod.yml +++ b/.github/workflows/deploy-preprod.yml @@ -23,16 +23,19 @@ jobs: aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: eu-west-1 - - name: 'Get Previous tag' - id: previoustag - uses: "WyriHaximus/github-action-get-previous-tag@v1" + - name: Get the latest budibase release version + id: version + run: | + release_version=$(cat lerna.json | jq -r '.version') + echo "RELEASE_VERSION=$release_version" >> $GITHUB_ENV - # - name: Pull values.yaml from budibase-infra - # run: | - # curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \ [c3a7a9d12] - # -H 'Accept: application/vnd.github.v3.raw' \ - # -o values.preprod.yaml \ - # -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/values.preprod.yaml + - name: Pull values.yaml from budibase-infra + run: | + curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \ + -H 'Accept: application/vnd.github.v3.raw' \ + -o values.preprod.yaml \ + -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/budibase-preprod/values.yaml + wc -l values.preprod.yaml - name: Deploy to Preprod Environment uses: deliverybot/helm@v1 @@ -41,13 +44,17 @@ jobs: namespace: budibase chart: charts/budibase token: ${{ github.token }} + helm: helm3 values: | globals: - appVersion: ${{ steps.previoustag.outputs.tag }} - # value-files: >- - # [ - # "charts/budibase/values.yaml" - # ] + appVersion: v${{ env.RELEASE_VERSION }} + ingress: + enabled: true + nginx: true + value-files: >- + [ + "values.preprod.yaml" + ] env: KUBECONFIG_FILE: '${{ secrets.PREPROD_KUBECONFIG }}' diff --git a/.github/workflows/release-develop.yml b/.github/workflows/release-develop.yml index b0f052f68..4467cd6c8 100644 --- a/.github/workflows/release-develop.yml +++ b/.github/workflows/release-develop.yml @@ -4,6 +4,16 @@ on: push: branches: - develop + paths: + - '.aws/**' + - '.github/**' + - 'charts/**' + - 'packages/**' + - 'scripts/**' + - 'package.json' + - 'yarn.lock' + - 'package.json' + - 'yarn.lock' env: POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }} diff --git a/.github/workflows/release-selfhost.yml b/.github/workflows/release-selfhost.yml index 31af7fe57..e842da9d7 100644 --- a/.github/workflows/release-selfhost.yml +++ b/.github/workflows/release-selfhost.yml @@ -9,12 +9,12 @@ jobs: steps: - uses: actions/checkout@v2 - - uses: actions/setup-node@v1 - with: + with: node-version: 14.x + fetch_depth: 0 - name: Tag and release Docker images (Self Host) - run: | + run: | docker login -u $DOCKER_USER -p $DOCKER_PASSWORD # Get latest release version @@ -29,7 +29,7 @@ jobs: # Tag apps and worker images docker tag budibase/apps:$release_tag budibase/apps:$SELFHOST_TAG docker tag budibase/worker:$release_tag budibase/worker:$SELFHOST_TAG - + # Push images docker push budibase/apps:$SELFHOST_TAG docker push budibase/worker:$SELFHOST_TAG @@ -37,37 +37,32 @@ jobs: DOCKER_USER: ${{ secrets.DOCKER_USERNAME }} DOCKER_PASSWORD: ${{ secrets.DOCKER_API_KEY }} SELFHOST_TAG: latest - - - name: Setup Helm - uses: azure/setup-helm@v1 - id: helm-install - + - name: Build CLI executables run: | pushd packages/cli yarn yarn build popd + + - name: Setup Helm + uses: azure/setup-helm@v1 + id: helm-install - # - name: Build and release helm chart - # run: | - # git config user.name "Budibase Helm Bot" - # git config user.email "<>" - # mv budibase-${{ env.RELEASE_VERSION }}.tgz docs - # helm repo index docs - # git checkout gh-pages - # git add -A - # git commit -m "Helm Release: ${{ env.RELEASE_VERSION }}" - # git push - # env: - # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - - # - name: Deploy - # uses: peaceiris/actions-gh-pages@v3 - # with: - # github_token: ${{ secrets.GITHUB_TOKEN }} - # publish_dir: ./public - # full_commit_message: "Helm Release: ${{ env.RELEASE_VERSION }}" + - name: Build and release helm chart + run: | + git config user.name "Budibase Helm Bot" + git config user.email "<>" + git pull + helm package charts/budibase + git checkout gh-pages + mv *.tgz docs + helm repo index docs + git add -A + git commit -m "Helm Release: ${{ env.RELEASE_VERSION }}" + git push + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Perform Github Release uses: softprops/action-gh-release@v1 @@ -78,4 +73,4 @@ jobs: files: | packages/cli/build/cli-win.exe packages/cli/build/cli-linux - packages/cli/build/cli-macos \ No newline at end of file + packages/cli/build/cli-macos \ No newline at end of file diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 859d89f18..82848c78e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -3,7 +3,17 @@ name: Budibase Release on: push: branches: - - test + - master + paths: + - '.aws/**' + - '.github/**' + - 'charts/**' + - 'packages/**' + - 'scripts/**' + - 'package.json' + - 'yarn.lock' + - 'package.json' + - 'yarn.lock' env: POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }} @@ -56,34 +66,3 @@ jobs: DOCKER_USER: ${{ secrets.DOCKER_USERNAME }} DOCKER_PASSWORD: ${{ secrets.DOCKER_API_KEY }} BUDIBASE_RELEASE_VERSION: ${{ steps.previoustag.outputs.tag }} - - # - name: Pull values.yaml from budibase-infra - # run: | - # curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \ [c3a7a9d12] - # -H 'Accept: application/vnd.github.v3.raw' \ - # -o values.preprod.yaml \ - # -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/values.preprod.yaml - - # - name: Deploy to Preprod Environment - # uses: deliverybot/helm@v1 - # with: - # release: budibase-preprod - # namespace: budibase - # chart: charts/budibase - # token: ${{ github.token }} - # values: | - # globals: - # appVersion: ${{ steps.previoustag.outputs.tag }} - # # value-files: >- - # # [ - # # "charts/budibase/values.yaml" - # # ] - # env: - # KUBECONFIG_FILE: '${{ secrets.PREPROD_KUBECONFIG }}' - - # - name: Discord Webhook Action - # uses: tsickert/discord-webhook@v4.0.0 - # with: - # webhook-url: ${{ secrets.PROD_DEPLOY_WEBHOOK_URL }} - # content: "Preprod Deployment Complete: ${{ env.RELEASE_VERSION }} deployed to Budibase Pre-prod." - # embed-title: ${{ env.RELEASE_VERSION }} diff --git a/charts/budibase/Chart.lock b/charts/budibase/Chart.lock new file mode 100644 index 000000000..75b9de07b --- /dev/null +++ b/charts/budibase/Chart.lock @@ -0,0 +1,9 @@ +dependencies: +- name: couchdb + repository: https://apache.github.io/couchdb-helm + version: 3.3.4 +- name: ingress-nginx + repository: https://kubernetes.github.io/ingress-nginx + version: 4.0.13 +digest: sha256:20892705c2d8e64c98257d181063a514ac55013e2b43399a6e54868a97f97845 +generated: "2021-12-30T18:55:30.878411Z" diff --git a/charts/budibase/Chart.yaml b/charts/budibase/Chart.yaml index cec5af291..8c9d44f20 100644 --- a/charts/budibase/Chart.yaml +++ b/charts/budibase/Chart.yaml @@ -11,14 +11,14 @@ sources: - https://github.com/Budibase/budibase - https://budibase.com type: application -version: 1.0.0 -appVersion: 1.0.20 +version: 0.2.5 +appVersion: 1.0.25 dependencies: - name: couchdb version: 3.3.4 repository: https://apache.github.io/couchdb-helm condition: services.couchdb.enabled - name: ingress-nginx - version: 3.35.0 - repository: https://github.com/kubernetes/ingress-nginx + version: 4.0.13 + repository: https://kubernetes.github.io/ingress-nginx condition: ingress.nginx diff --git a/charts/budibase/charts/ingress-nginx-3.35.0.tgz b/charts/budibase/charts/ingress-nginx-3.35.0.tgz deleted file mode 100644 index ee5214c49..000000000 Binary files a/charts/budibase/charts/ingress-nginx-3.35.0.tgz and /dev/null differ diff --git a/charts/budibase/charts/ingress-nginx-4.0.13.tgz b/charts/budibase/charts/ingress-nginx-4.0.13.tgz new file mode 100644 index 000000000..1e34215c5 Binary files /dev/null and b/charts/budibase/charts/ingress-nginx-4.0.13.tgz differ diff --git a/charts/budibase/templates/proxy-service-service.yaml b/charts/budibase/templates/proxy-service-service.yaml index 8f14d9786..bf2b199ee 100644 --- a/charts/budibase/templates/proxy-service-service.yaml +++ b/charts/budibase/templates/proxy-service-service.yaml @@ -9,12 +9,11 @@ metadata: app.kubernetes.io/name: budibase-proxy name: proxy-service spec: - type: NodePort ports: - - port: {{ .Values.services.proxy.port }} + - name: {{ .Values.services.proxy.port | quote }} + port: {{ .Values.services.proxy.port }} targetPort: {{ .Values.services.proxy.port }} - protocol: TCP selector: app.kubernetes.io/name: budibase-proxy status: - loadBalancer: {} + loadBalancer: {} \ No newline at end of file diff --git a/charts/budibase/values.yaml b/charts/budibase/values.yaml index b634f1581..4666d01c7 100644 --- a/charts/budibase/values.yaml +++ b/charts/budibase/values.yaml @@ -40,7 +40,7 @@ service: port: 10000 ingress: - enabled: false + enabled: true aws: false nginx: true certificateArn: "" @@ -302,4 +302,4 @@ couchdb: initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 - timeoutSeconds: 1 \ No newline at end of file + timeoutSeconds: 1 diff --git a/lerna.json b/lerna.json index b77de3276..4bbdf26b2 100644 --- a/lerna.json +++ b/lerna.json @@ -1,5 +1,5 @@ { - "version": "1.0.27-alpha.6", + "version": "1.0.27-alpha.17", "npmClient": "yarn", "packages": [ "packages/*" diff --git a/packages/auth/.gitignore b/packages/backend-core/.gitignore similarity index 100% rename from packages/auth/.gitignore rename to packages/backend-core/.gitignore diff --git a/packages/auth/LICENSE b/packages/backend-core/LICENSE similarity index 100% rename from packages/auth/LICENSE rename to packages/backend-core/LICENSE diff --git a/packages/auth/README.md b/packages/backend-core/README.md similarity index 78% rename from packages/auth/README.md rename to packages/backend-core/README.md index 4c6a474b5..f36f80f84 100644 --- a/packages/auth/README.md +++ b/packages/backend-core/README.md @@ -4,7 +4,7 @@ This library contains core functionality, like auth and security features which are shared between backend services. #### Note about top level JS files -For the purposes of being able to do say `require("@budibase/auth/permissions")` we need to +For the purposes of being able to do say `require("@budibase/backend-core/permissions")` we need to specify the exports at the top-level of the module. For these files they should be limited to a single `require` of the file that should diff --git a/packages/auth/accounts.js b/packages/backend-core/accounts.js similarity index 100% rename from packages/auth/accounts.js rename to packages/backend-core/accounts.js diff --git a/packages/backend-core/auth.js b/packages/backend-core/auth.js new file mode 100644 index 000000000..bbfe3d41d --- /dev/null +++ b/packages/backend-core/auth.js @@ -0,0 +1 @@ +module.exports = require("./src/auth") diff --git a/packages/auth/cache.js b/packages/backend-core/cache.js similarity index 100% rename from packages/auth/cache.js rename to packages/backend-core/cache.js diff --git a/packages/auth/constants.js b/packages/backend-core/constants.js similarity index 100% rename from packages/auth/constants.js rename to packages/backend-core/constants.js diff --git a/packages/auth/db.js b/packages/backend-core/db.js similarity index 100% rename from packages/auth/db.js rename to packages/backend-core/db.js diff --git a/packages/auth/deprovision.js b/packages/backend-core/deprovision.js similarity index 100% rename from packages/auth/deprovision.js rename to packages/backend-core/deprovision.js diff --git a/packages/backend-core/middleware.js b/packages/backend-core/middleware.js new file mode 100644 index 000000000..30fec9623 --- /dev/null +++ b/packages/backend-core/middleware.js @@ -0,0 +1 @@ +module.exports = require("./src/middleware") diff --git a/packages/backend-core/objectStore.js b/packages/backend-core/objectStore.js new file mode 100644 index 000000000..3ee433f22 --- /dev/null +++ b/packages/backend-core/objectStore.js @@ -0,0 +1,4 @@ +module.exports = { + ...require("./src/objectStore"), + ...require("./src/objectStore/utils"), +} diff --git a/packages/auth/package.json b/packages/backend-core/package.json similarity index 87% rename from packages/auth/package.json rename to packages/backend-core/package.json index a0e169486..648a49371 100644 --- a/packages/auth/package.json +++ b/packages/backend-core/package.json @@ -1,7 +1,7 @@ { - "name": "@budibase/auth", - "version": "1.0.27-alpha.6", - "description": "Authentication middlewares for budibase builder and apps", + "name": "@budibase/backend-core", + "version": "1.0.27-alpha.17", + "description": "Budibase backend core libraries used in server and worker", "main": "src/index.js", "author": "Budibase", "license": "GPL-3.0", diff --git a/packages/auth/permissions.js b/packages/backend-core/permissions.js similarity index 100% rename from packages/auth/permissions.js rename to packages/backend-core/permissions.js diff --git a/packages/auth/redis.js b/packages/backend-core/redis.js similarity index 100% rename from packages/auth/redis.js rename to packages/backend-core/redis.js diff --git a/packages/auth/roles.js b/packages/backend-core/roles.js similarity index 100% rename from packages/auth/roles.js rename to packages/backend-core/roles.js diff --git a/packages/auth/scripts/jestSetup.js b/packages/backend-core/scripts/jestSetup.js similarity index 100% rename from packages/auth/scripts/jestSetup.js rename to packages/backend-core/scripts/jestSetup.js diff --git a/packages/auth/sessions.js b/packages/backend-core/sessions.js similarity index 100% rename from packages/auth/sessions.js rename to packages/backend-core/sessions.js diff --git a/packages/auth/src/index.js b/packages/backend-core/src/auth.js similarity index 52% rename from packages/auth/src/index.js rename to packages/backend-core/src/auth.js index 4aa2c8ab9..7f66d887a 100644 --- a/packages/auth/src/index.js +++ b/packages/backend-core/src/auth.js @@ -1,7 +1,6 @@ const passport = require("koa-passport") const LocalStrategy = require("passport-local").Strategy const JwtStrategy = require("passport-jwt").Strategy -const { StaticDatabases } = require("./db/utils") const { getGlobalDB } = require("./tenancy") const { jwt, @@ -14,8 +13,6 @@ const { appTenancy, authError, } = require("./middleware") -const { setDB } = require("./db") -const userCache = require("./cache/user") // Strategies passport.use(new LocalStrategy(local.options, local.authenticate)) @@ -36,36 +33,13 @@ passport.deserializeUser(async (user, done) => { }) module.exports = { - init(pouch) { - setDB(pouch) - }, - db: require("./db/utils"), - redis: { - Client: require("./redis"), - utils: require("./redis/utils"), - }, - objectStore: { - ...require("./objectStore"), - ...require("./objectStore/utils"), - }, - utils: { - ...require("./utils"), - ...require("./hashing"), - }, - auth: { - buildAuthMiddleware: authenticated, - passport, - google, - oidc, - jwt: require("jsonwebtoken"), - buildTenancyMiddleware: tenancy, - buildAppTenancyMiddleware: appTenancy, - auditLog, - authError, - }, - cache: { - user: userCache, - }, - StaticDatabases, - constants: require("./constants"), + buildAuthMiddleware: authenticated, + passport, + google, + oidc, + jwt: require("jsonwebtoken"), + buildTenancyMiddleware: tenancy, + buildAppTenancyMiddleware: appTenancy, + auditLog, + authError, } diff --git a/packages/auth/src/cache/appMetadata.js b/packages/backend-core/src/cache/appMetadata.js similarity index 100% rename from packages/auth/src/cache/appMetadata.js rename to packages/backend-core/src/cache/appMetadata.js diff --git a/packages/auth/src/cache/user.js b/packages/backend-core/src/cache/user.js similarity index 100% rename from packages/auth/src/cache/user.js rename to packages/backend-core/src/cache/user.js diff --git a/packages/auth/src/cloud/accounts.js b/packages/backend-core/src/cloud/accounts.js similarity index 100% rename from packages/auth/src/cloud/accounts.js rename to packages/backend-core/src/cloud/accounts.js diff --git a/packages/auth/src/cloud/api.js b/packages/backend-core/src/cloud/api.js similarity index 100% rename from packages/auth/src/cloud/api.js rename to packages/backend-core/src/cloud/api.js diff --git a/packages/auth/src/constants.js b/packages/backend-core/src/constants.js similarity index 100% rename from packages/auth/src/constants.js rename to packages/backend-core/src/constants.js diff --git a/packages/auth/src/db/Replication.js b/packages/backend-core/src/db/Replication.js similarity index 100% rename from packages/auth/src/db/Replication.js rename to packages/backend-core/src/db/Replication.js diff --git a/packages/auth/src/db/constants.js b/packages/backend-core/src/db/constants.js similarity index 100% rename from packages/auth/src/db/constants.js rename to packages/backend-core/src/db/constants.js diff --git a/packages/auth/src/db/index.js b/packages/backend-core/src/db/index.js similarity index 100% rename from packages/auth/src/db/index.js rename to packages/backend-core/src/db/index.js diff --git a/packages/auth/src/db/utils.js b/packages/backend-core/src/db/utils.js similarity index 97% rename from packages/auth/src/db/utils.js rename to packages/backend-core/src/db/utils.js index 2bad09c30..5830de472 100644 --- a/packages/auth/src/db/utils.js +++ b/packages/backend-core/src/db/utils.js @@ -224,8 +224,15 @@ exports.getAllDbs = async () => { } } let couchUrl = `${exports.getCouchUrl()}/_all_dbs` - if (env.MULTI_TENANCY) { - let tenantId = getTenantId() + let tenantId = getTenantId() + if (!env.MULTI_TENANCY || tenantId == DEFAULT_TENANT_ID) { + // just get all DBs when: + // - single tenancy + // - default tenant + // - apps dbs don't contain tenant id + // - non-default tenant dbs are filtered out application side in getAllApps + await addDbs(couchUrl) + } else { // get prod apps await addDbs( exports.getStartEndKeyURL(couchUrl, DocumentTypes.APP, tenantId) @@ -236,9 +243,6 @@ exports.getAllDbs = async () => { ) // add global db name dbs.push(getGlobalDBName(tenantId)) - } else { - // just get all DBs in self host - await addDbs(couchUrl) } return dbs } diff --git a/packages/auth/src/db/views.js b/packages/backend-core/src/db/views.js similarity index 100% rename from packages/auth/src/db/views.js rename to packages/backend-core/src/db/views.js diff --git a/packages/auth/src/environment.js b/packages/backend-core/src/environment.js similarity index 100% rename from packages/auth/src/environment.js rename to packages/backend-core/src/environment.js diff --git a/packages/auth/src/hashing.js b/packages/backend-core/src/hashing.js similarity index 100% rename from packages/auth/src/hashing.js rename to packages/backend-core/src/hashing.js diff --git a/packages/auth/src/helpers.js b/packages/backend-core/src/helpers.js similarity index 100% rename from packages/auth/src/helpers.js rename to packages/backend-core/src/helpers.js diff --git a/packages/backend-core/src/index.js b/packages/backend-core/src/index.js new file mode 100644 index 000000000..cd3a3f5c9 --- /dev/null +++ b/packages/backend-core/src/index.js @@ -0,0 +1,17 @@ +const { setDB } = require("./db") + +module.exports = { + init(pouch) { + setDB(pouch) + }, + // some default exports from the library, however these ideally shouldn't + // be used, instead the syntax require("@budibase/backend-core/db") should be used + StaticDatabases: require("./db/utils").StaticDatabases, + db: require("../db"), + redis: require("../redis"), + objectStore: require("../objectStore"), + utils: require("../utils"), + cache: require("../cache"), + auth: require("../auth"), + constants: require("../constants"), +} diff --git a/packages/auth/src/middleware/appTenancy.js b/packages/backend-core/src/middleware/appTenancy.js similarity index 100% rename from packages/auth/src/middleware/appTenancy.js rename to packages/backend-core/src/middleware/appTenancy.js diff --git a/packages/auth/src/middleware/auditLog.js b/packages/backend-core/src/middleware/auditLog.js similarity index 100% rename from packages/auth/src/middleware/auditLog.js rename to packages/backend-core/src/middleware/auditLog.js diff --git a/packages/auth/src/middleware/authenticated.js b/packages/backend-core/src/middleware/authenticated.js similarity index 100% rename from packages/auth/src/middleware/authenticated.js rename to packages/backend-core/src/middleware/authenticated.js diff --git a/packages/auth/src/middleware/index.js b/packages/backend-core/src/middleware/index.js similarity index 100% rename from packages/auth/src/middleware/index.js rename to packages/backend-core/src/middleware/index.js diff --git a/packages/auth/src/middleware/matchers.js b/packages/backend-core/src/middleware/matchers.js similarity index 100% rename from packages/auth/src/middleware/matchers.js rename to packages/backend-core/src/middleware/matchers.js diff --git a/packages/auth/src/middleware/passport/google.js b/packages/backend-core/src/middleware/passport/google.js similarity index 100% rename from packages/auth/src/middleware/passport/google.js rename to packages/backend-core/src/middleware/passport/google.js diff --git a/packages/auth/src/middleware/passport/jwt.js b/packages/backend-core/src/middleware/passport/jwt.js similarity index 100% rename from packages/auth/src/middleware/passport/jwt.js rename to packages/backend-core/src/middleware/passport/jwt.js diff --git a/packages/auth/src/middleware/passport/local.js b/packages/backend-core/src/middleware/passport/local.js similarity index 100% rename from packages/auth/src/middleware/passport/local.js rename to packages/backend-core/src/middleware/passport/local.js diff --git a/packages/auth/src/middleware/passport/oidc.js b/packages/backend-core/src/middleware/passport/oidc.js similarity index 100% rename from packages/auth/src/middleware/passport/oidc.js rename to packages/backend-core/src/middleware/passport/oidc.js diff --git a/packages/auth/src/middleware/passport/tests/google.spec.js b/packages/backend-core/src/middleware/passport/tests/google.spec.js similarity index 100% rename from packages/auth/src/middleware/passport/tests/google.spec.js rename to packages/backend-core/src/middleware/passport/tests/google.spec.js diff --git a/packages/auth/src/middleware/passport/tests/oidc.spec.js b/packages/backend-core/src/middleware/passport/tests/oidc.spec.js similarity index 100% rename from packages/auth/src/middleware/passport/tests/oidc.spec.js rename to packages/backend-core/src/middleware/passport/tests/oidc.spec.js diff --git a/packages/auth/src/middleware/passport/tests/third-party-common.spec.js b/packages/backend-core/src/middleware/passport/tests/third-party-common.spec.js similarity index 100% rename from packages/auth/src/middleware/passport/tests/third-party-common.spec.js rename to packages/backend-core/src/middleware/passport/tests/third-party-common.spec.js diff --git a/packages/auth/src/middleware/passport/tests/utilities/mock-data.js b/packages/backend-core/src/middleware/passport/tests/utilities/mock-data.js similarity index 100% rename from packages/auth/src/middleware/passport/tests/utilities/mock-data.js rename to packages/backend-core/src/middleware/passport/tests/utilities/mock-data.js diff --git a/packages/auth/src/middleware/passport/third-party-common.js b/packages/backend-core/src/middleware/passport/third-party-common.js similarity index 100% rename from packages/auth/src/middleware/passport/third-party-common.js rename to packages/backend-core/src/middleware/passport/third-party-common.js diff --git a/packages/auth/src/middleware/passport/utils.js b/packages/backend-core/src/middleware/passport/utils.js similarity index 100% rename from packages/auth/src/middleware/passport/utils.js rename to packages/backend-core/src/middleware/passport/utils.js diff --git a/packages/auth/src/middleware/tenancy.js b/packages/backend-core/src/middleware/tenancy.js similarity index 100% rename from packages/auth/src/middleware/tenancy.js rename to packages/backend-core/src/middleware/tenancy.js diff --git a/packages/auth/src/migrations/index.js b/packages/backend-core/src/migrations/index.js similarity index 100% rename from packages/auth/src/migrations/index.js rename to packages/backend-core/src/migrations/index.js diff --git a/packages/auth/src/migrations/tests/__snapshots__/index.spec.js.snap b/packages/backend-core/src/migrations/tests/__snapshots__/index.spec.js.snap similarity index 100% rename from packages/auth/src/migrations/tests/__snapshots__/index.spec.js.snap rename to packages/backend-core/src/migrations/tests/__snapshots__/index.spec.js.snap diff --git a/packages/auth/src/migrations/tests/index.spec.js b/packages/backend-core/src/migrations/tests/index.spec.js similarity index 100% rename from packages/auth/src/migrations/tests/index.spec.js rename to packages/backend-core/src/migrations/tests/index.spec.js diff --git a/packages/auth/src/objectStore/index.js b/packages/backend-core/src/objectStore/index.js similarity index 91% rename from packages/auth/src/objectStore/index.js rename to packages/backend-core/src/objectStore/index.js index 87b67d464..b5d8475ce 100644 --- a/packages/auth/src/objectStore/index.js +++ b/packages/backend-core/src/objectStore/index.js @@ -206,6 +206,34 @@ exports.retrieveToTmp = async (bucketName, filepath) => { return outputPath } +/** + * Delete a single file. + */ +exports.deleteFile = async (bucketName, filepath) => { + const objectStore = exports.ObjectStore(bucketName) + await exports.makeSureBucketExists(objectStore, bucketName) + const params = { + Bucket: bucketName, + Key: filepath, + } + return objectStore.deleteObject(params) +} + +exports.deleteFiles = async (bucketName, filepaths) => { + const objectStore = exports.ObjectStore(bucketName) + await exports.makeSureBucketExists(objectStore, bucketName) + const params = { + Bucket: bucketName, + Delete: { + Objects: filepaths.map(path => ({ Key: path })), + }, + } + return objectStore.deleteObjects(params).promise() +} + +/** + * Delete a path, including everything within. + */ exports.deleteFolder = async (bucketName, folder) => { bucketName = sanitizeBucket(bucketName) folder = sanitizeKey(folder) diff --git a/packages/auth/src/objectStore/utils.js b/packages/backend-core/src/objectStore/utils.js similarity index 100% rename from packages/auth/src/objectStore/utils.js rename to packages/backend-core/src/objectStore/utils.js diff --git a/packages/auth/src/redis/authRedis.js b/packages/backend-core/src/redis/authRedis.js similarity index 100% rename from packages/auth/src/redis/authRedis.js rename to packages/backend-core/src/redis/authRedis.js diff --git a/packages/auth/src/redis/index.js b/packages/backend-core/src/redis/index.js similarity index 100% rename from packages/auth/src/redis/index.js rename to packages/backend-core/src/redis/index.js diff --git a/packages/auth/src/redis/utils.js b/packages/backend-core/src/redis/utils.js similarity index 100% rename from packages/auth/src/redis/utils.js rename to packages/backend-core/src/redis/utils.js diff --git a/packages/auth/src/security/permissions.js b/packages/backend-core/src/security/permissions.js similarity index 100% rename from packages/auth/src/security/permissions.js rename to packages/backend-core/src/security/permissions.js diff --git a/packages/auth/src/security/roles.js b/packages/backend-core/src/security/roles.js similarity index 100% rename from packages/auth/src/security/roles.js rename to packages/backend-core/src/security/roles.js diff --git a/packages/auth/src/security/sessions.js b/packages/backend-core/src/security/sessions.js similarity index 100% rename from packages/auth/src/security/sessions.js rename to packages/backend-core/src/security/sessions.js diff --git a/packages/auth/src/tenancy/FunctionContext.js b/packages/backend-core/src/tenancy/FunctionContext.js similarity index 100% rename from packages/auth/src/tenancy/FunctionContext.js rename to packages/backend-core/src/tenancy/FunctionContext.js diff --git a/packages/auth/src/tenancy/context.js b/packages/backend-core/src/tenancy/context.js similarity index 100% rename from packages/auth/src/tenancy/context.js rename to packages/backend-core/src/tenancy/context.js diff --git a/packages/auth/src/tenancy/deprovision.js b/packages/backend-core/src/tenancy/deprovision.js similarity index 100% rename from packages/auth/src/tenancy/deprovision.js rename to packages/backend-core/src/tenancy/deprovision.js diff --git a/packages/auth/src/tenancy/index.js b/packages/backend-core/src/tenancy/index.js similarity index 100% rename from packages/auth/src/tenancy/index.js rename to packages/backend-core/src/tenancy/index.js diff --git a/packages/auth/src/tenancy/tenancy.js b/packages/backend-core/src/tenancy/tenancy.js similarity index 100% rename from packages/auth/src/tenancy/tenancy.js rename to packages/backend-core/src/tenancy/tenancy.js diff --git a/packages/auth/src/tests/utilities/db.js b/packages/backend-core/src/tests/utilities/db.js similarity index 100% rename from packages/auth/src/tests/utilities/db.js rename to packages/backend-core/src/tests/utilities/db.js diff --git a/packages/auth/src/tests/utilities/dbConfig.js b/packages/backend-core/src/tests/utilities/dbConfig.js similarity index 100% rename from packages/auth/src/tests/utilities/dbConfig.js rename to packages/backend-core/src/tests/utilities/dbConfig.js diff --git a/packages/auth/src/utils.js b/packages/backend-core/src/utils.js similarity index 100% rename from packages/auth/src/utils.js rename to packages/backend-core/src/utils.js diff --git a/packages/auth/tenancy.js b/packages/backend-core/tenancy.js similarity index 100% rename from packages/auth/tenancy.js rename to packages/backend-core/tenancy.js diff --git a/packages/backend-core/utils.js b/packages/backend-core/utils.js new file mode 100644 index 000000000..2ef920e10 --- /dev/null +++ b/packages/backend-core/utils.js @@ -0,0 +1,4 @@ +module.exports = { + ...require("./src/utils"), + ...require("./src/hashing"), +} diff --git a/packages/auth/yarn.lock b/packages/backend-core/yarn.lock similarity index 100% rename from packages/auth/yarn.lock rename to packages/backend-core/yarn.lock diff --git a/packages/bbui/package.json b/packages/bbui/package.json index 60be1657a..199ccd6db 100644 --- a/packages/bbui/package.json +++ b/packages/bbui/package.json @@ -1,7 +1,7 @@ { "name": "@budibase/bbui", "description": "A UI solution used in the different Budibase projects.", - "version": "1.0.27-alpha.6", + "version": "1.0.27-alpha.17", "license": "MPL-2.0", "svelte": "src/index.js", "module": "dist/bbui.es.js", diff --git a/packages/bbui/src/Tabs/Tabs.svelte b/packages/bbui/src/Tabs/Tabs.svelte index c68f715de..40e02058c 100644 --- a/packages/bbui/src/Tabs/Tabs.svelte +++ b/packages/bbui/src/Tabs/Tabs.svelte @@ -10,6 +10,8 @@ export let noHorizPadding = false export let quiet = false export let emphasized = false + // overlay content from the tab bar onto tabs e.g. for a dropdown + export let onTop = false let thisSelected = undefined @@ -78,6 +80,7 @@ 'spectrum-Tabs--quiet'} spectrum-Tabs--{vertical ? 'vertical' : 'horizontal'}" + class:onTop > {#if $tab.info} @@ -98,7 +101,9 @@ .quiet { border-bottom: none !important; } - + .onTop { + z-index: 20; + } .spectrum-Tabs { padding-left: var(--spacing-xl); padding-right: var(--spacing-xl); diff --git a/packages/bbui/yarn.lock b/packages/bbui/yarn.lock index a102e6c14..42c88af5a 100644 --- a/packages/bbui/yarn.lock +++ b/packages/bbui/yarn.lock @@ -2076,9 +2076,9 @@ postcss@^7.0.0, postcss@^7.0.1, postcss@^7.0.27: supports-color "^6.1.0" postcss@^8.2.9: - version "8.2.10" - resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.2.10.tgz#ca7a042aa8aff494b334d0ff3e9e77079f6f702b" - integrity sha512-b/h7CPV7QEdrqIxtAf2j31U5ef05uBDuvoXv6L51Q4rcS1jdlXAVKJv+atCFdUXYl9dyTHGyoMzIepwowRJjFw== + version "8.2.13" + resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.2.13.tgz#dbe043e26e3c068e45113b1ed6375d2d37e2129f" + integrity sha512-FCE5xLH+hjbzRdpbRb1IMCvPv9yZx2QnDarBEYSN0N0HYk+TcXsEhwdFcFb+SRWOKzKGErhIEbBK2ogyLdTtfQ== dependencies: colorette "^1.2.2" nanoid "^3.1.22" diff --git a/packages/builder/package.json b/packages/builder/package.json index e2b3e4e2f..06583ffde 100644 --- a/packages/builder/package.json +++ b/packages/builder/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/builder", - "version": "1.0.27-alpha.6", + "version": "1.0.27-alpha.17", "license": "GPL-3.0", "private": true, "scripts": { @@ -65,10 +65,10 @@ } }, "dependencies": { - "@budibase/bbui": "^1.0.27-alpha.6", - "@budibase/client": "^1.0.27-alpha.6", + "@budibase/bbui": "^1.0.27-alpha.17", + "@budibase/client": "^1.0.27-alpha.17", "@budibase/colorpicker": "1.1.2", - "@budibase/string-templates": "^1.0.27-alpha.6", + "@budibase/string-templates": "^1.0.27-alpha.17", "@sentry/browser": "5.19.1", "@spectrum-css/page": "^3.0.1", "@spectrum-css/vars": "^3.0.1", diff --git a/packages/builder/src/builderStore/datasource.js b/packages/builder/src/builderStore/datasource.js index 61cead2e1..cfdeeac23 100644 --- a/packages/builder/src/builderStore/datasource.js +++ b/packages/builder/src/builderStore/datasource.js @@ -23,10 +23,10 @@ function prepareData(config) { return datasource } -export async function saveDatasource(config) { +export async function saveDatasource(config, skipFetch = false) { const datasource = prepareData(config) // Create datasource - const resp = await datasources.save(datasource, datasource.plus) + const resp = await datasources.save(datasource, !skipFetch && datasource.plus) // update the tables incase data source plus await tables.fetch() diff --git a/packages/builder/src/components/automation/AutomationBuilder/FlowChart/FlowItem.svelte b/packages/builder/src/components/automation/AutomationBuilder/FlowChart/FlowItem.svelte index 7f379ba13..fe94b7e63 100644 --- a/packages/builder/src/components/automation/AutomationBuilder/FlowChart/FlowItem.svelte +++ b/packages/builder/src/components/automation/AutomationBuilder/FlowChart/FlowItem.svelte @@ -9,6 +9,7 @@ Modal, Button, StatusLight, + ActionButton, } from "@budibase/bbui" import AutomationBlockSetup from "../../SetupPanel/AutomationBlockSetup.svelte" import CreateWebhookModal from "components/automation/Shared/CreateWebhookModal.svelte" @@ -27,7 +28,7 @@ let blockComplete $: testResult = $automationStore.selectedAutomation.testResults?.steps.filter( - step => step.stepId === block.stepId + step => (block.id ? step.id === block.id : step.stepId === block.stepId) ) $: isTrigger = block.type === "TRIGGER" @@ -119,19 +120,13 @@
-
{ - setupToggled = !setupToggled - }} - class="center-items" + (setupToggled = !setupToggled)} + quiet + icon={setupToggled ? "ChevronDown" : "ChevronRight"} > - {#if setupToggled} - - {:else} - - {/if} Setup -
+ {#if !isTrigger}
deleteStep()}> @@ -187,6 +182,7 @@ .splitHeader { display: flex; justify-content: space-between; + align-items: center; } .iconAlign { padding: 0 0 0 var(--spacing-m); diff --git a/packages/builder/src/components/automation/SetupPanel/AutomationBlockSetup.svelte b/packages/builder/src/components/automation/SetupPanel/AutomationBlockSetup.svelte index d648b3f98..928897d6f 100644 --- a/packages/builder/src/components/automation/SetupPanel/AutomationBlockSetup.svelte +++ b/packages/builder/src/components/automation/SetupPanel/AutomationBlockSetup.svelte @@ -10,6 +10,7 @@ ActionButton, Drawer, Modal, + Detail, } from "@budibase/bbui" import CreateWebhookModal from "components/automation/Shared/CreateWebhookModal.svelte" @@ -37,6 +38,7 @@ let drawer let tempFilters = lookForFilters(schemaProperties) || [] let fillWidth = true + let codeBindingOpen = false $: stepId = block.stepId $: bindings = getAvailableBindings( @@ -233,7 +235,16 @@ onChange(e, key)} value={inputData[key]} /> {:else if value.customType === "code"} - 
{JSON.stringify(bindings, null, 2)}
+ (codeBindingOpen = !codeBindingOpen)} + quiet + icon={codeBindingOpen ? "ChevronDown" : "ChevronRight"} + > + Bindings + + {#if codeBindingOpen} + 
{JSON.stringify(bindings, null, 2)}
+ {/if} { diff --git a/packages/builder/src/components/backend/DatasourceNavigator/DatasourceNavigator.svelte b/packages/builder/src/components/backend/DatasourceNavigator/DatasourceNavigator.svelte index 18143c207..af345ddcd 100644 --- a/packages/builder/src/components/backend/DatasourceNavigator/DatasourceNavigator.svelte +++ b/packages/builder/src/components/backend/DatasourceNavigator/DatasourceNavigator.svelte @@ -137,7 +137,7 @@ selected={$queries.selected === query._id} on:click={() => onClickQuery(query)} > - + {/each} {/if} diff --git a/packages/builder/src/components/backend/DatasourceNavigator/TableIntegrationMenu/PlusConfigForm.svelte b/packages/builder/src/components/backend/DatasourceNavigator/TableIntegrationMenu/PlusConfigForm.svelte index b97f23d7a..819fb32e4 100644 --- a/packages/builder/src/components/backend/DatasourceNavigator/TableIntegrationMenu/PlusConfigForm.svelte +++ b/packages/builder/src/components/backend/DatasourceNavigator/TableIntegrationMenu/PlusConfigForm.svelte @@ -199,18 +199,18 @@ Tell budibase how your tables are related to get even more smart features. -{/if} -{#if relationshipInfo && relationshipInfo.length > 0} - openRelationshipModal(detail.from, detail.to)} - schema={relationshipSchema} - data={relationshipInfo} - allowEditColumns={false} - allowEditRows={false} - allowSelectRows={false} - /> -{:else} - No relationships configured. + {#if relationshipInfo && relationshipInfo.length > 0} +
openRelationshipModal(detail.from, detail.to)} + schema={relationshipSchema} + data={relationshipInfo} + allowEditColumns={false} + allowEditRows={false} + allowSelectRows={false} + /> + {:else} + No relationships configured. + {/if} {/if} diff --git a/packages/builder/src/components/integration/KeyValueBuilder.svelte b/packages/builder/src/components/integration/KeyValueBuilder.svelte index 94f3cb7f1..39eca0955 100644 --- a/packages/builder/src/components/integration/KeyValueBuilder.svelte +++ b/packages/builder/src/components/integration/KeyValueBuilder.svelte @@ -31,7 +31,10 @@ export let menuItems export let showMenu = false - let fields = Object.entries(object).map(([name, value]) => ({ name, value })) + let fields = Object.entries(object || {}).map(([name, value]) => ({ + name, + value, + })) let fieldActivity = buildFieldActivity(activity) $: object = fields.reduce( diff --git a/packages/builder/src/constants/backend/index.js b/packages/builder/src/constants/backend/index.js index a61f89430..c5fb294f8 100644 --- a/packages/builder/src/constants/backend/index.js +++ b/packages/builder/src/constants/backend/index.js @@ -219,3 +219,13 @@ export const RestBodyTypes = [ { name: "raw (XML)", value: "xml" }, { name: "raw (Text)", value: "text" }, ] + +export const PaginationTypes = [ + { label: "Page number based", value: "page" }, + { label: "Cursor based", value: "cursor" }, +] + +export const PaginationLocations = [ + { label: "Query parameters", value: "query" }, + { label: "Request body", value: "body" }, +] diff --git a/packages/builder/src/helpers/data/utils.js b/packages/builder/src/helpers/data/utils.js index b6478680e..0e9910918 100644 --- a/packages/builder/src/helpers/data/utils.js +++ b/packages/builder/src/helpers/data/utils.js @@ -84,7 +84,7 @@ export function customQueryIconText(datasource, query) { case "read": return "GET" case "delete": - return "DELETE" + return "DEL" case "patch": return "PATCH" } diff --git a/packages/builder/src/helpers/fetchTableData.js b/packages/builder/src/helpers/fetchTableData.js index 8b413f361..6d61ec813 100644 --- a/packages/builder/src/helpers/fetchTableData.js +++ b/packages/builder/src/helpers/fetchTableData.js @@ -1,5 +1,7 @@ // Do not use any aliased imports in common files, as these will be bundled -// by multiple bundlers which may not be able to resolve them +// by multiple bundlers which may not be able to resolve them. +// This will eventually be replaced by the new client implementation when we +// add a core package. import { writable, derived, get } from "svelte/store" import * as API from "../builderStore/api" import { buildLuceneQuery } from "./lucene" diff --git a/packages/builder/src/helpers/lucene.js b/packages/builder/src/helpers/lucene.js index 2348bb111..63fe542da 100644 --- a/packages/builder/src/helpers/lucene.js +++ b/packages/builder/src/helpers/lucene.js @@ -122,12 +122,16 @@ export const luceneQuery = (docs, query) => { // Process a string match (fails if the value does not start with the string) const stringMatch = match("string", (docValue, testValue) => { - return !docValue || !docValue.startsWith(testValue) + return ( + !docValue || !docValue?.toLowerCase().startsWith(testValue?.toLowerCase()) + ) }) // Process a fuzzy match (treat the same as starts with when running locally) const fuzzyMatch = match("fuzzy", (docValue, testValue) => { - return !docValue || !docValue.startsWith(testValue) + return ( + !docValue || !docValue?.toLowerCase().startsWith(testValue?.toLowerCase()) + ) }) // Process a range match diff --git a/packages/builder/src/pages/builder/app/[application]/data/datasource/[selectedDatasource]/rest/[query]/index.svelte b/packages/builder/src/pages/builder/app/[application]/data/datasource/[selectedDatasource]/rest/[query]/index.svelte index 43037242f..808c3a49e 100644 --- a/packages/builder/src/pages/builder/app/[application]/data/datasource/[selectedDatasource]/rest/[query]/index.svelte +++ b/packages/builder/src/pages/builder/app/[application]/data/datasource/[selectedDatasource]/rest/[query]/index.svelte @@ -30,6 +30,8 @@ import { RestBodyTypes as bodyTypes, SchemaTypeOptions, + PaginationLocations, + PaginationTypes, } from "constants/backend" import JSONPreview from "components/integration/JSONPreview.svelte" import AccessLevelSelect from "components/integration/AccessLevelSelect.svelte" @@ -232,8 +234,12 @@ const datasourceUrl = datasource?.config.url const qs = query?.fields.queryString breakQs = restUtils.breakQueryString(qs) - if (datasourceUrl && !query.fields.path?.startsWith(datasourceUrl)) { - const path = query.fields.path + const path = query.fields.path + if ( + datasourceUrl && + !path?.startsWith("http") && + !path?.startsWith("{{") // don't substitute the datasource url when query starts with a variable e.g. the upgrade path + ) { query.fields.path = `${datasource.config.url}/${path ? path : ""}` } url = buildUrl(query.fields.path, breakQs) @@ -265,6 +271,9 @@ query.fields.bodyType = RawRestBodyTypes.NONE } } + if (query && !query.fields.pagination) { + query.fields.pagination = {} + } dynamicVariables = getDynamicVariables(datasource, query._id) }) @@ -306,7 +315,7 @@ - + + + + {#if !$flags.queryTransformerBanner} @@ -450,7 +495,7 @@ Create dynamic variables based on response body or headers - from other queries. + from this query. diff --git a/packages/builder/src/stores/backend/queries.js b/packages/builder/src/stores/backend/queries.js index 0e767fa5c..2018933ff 100644 --- a/packages/builder/src/stores/backend/queries.js +++ b/packages/builder/src/stores/backend/queries.js @@ -134,7 +134,7 @@ export function createQueriesStore() { list.map(q => q.name) ) - actions.save(datasourceId, newQuery) + return actions.save(datasourceId, newQuery) }, } diff --git a/packages/cli/package.json b/packages/cli/package.json index 8f0867986..2f012e763 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/cli", - "version": "1.0.27-alpha.6", + "version": "1.0.27-alpha.17", "description": "Budibase CLI, for developers, self hosting and migrations.", "main": "src/index.js", "bin": { diff --git a/packages/client/package.json b/packages/client/package.json index bfe810502..15115c3c0 100644 --- a/packages/client/package.json +++ b/packages/client/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/client", - "version": "1.0.27-alpha.6", + "version": "1.0.27-alpha.17", "license": "MPL-2.0", "module": "dist/budibase-client.js", "main": "dist/budibase-client.js", @@ -19,9 +19,9 @@ "dev:builder": "rollup -cw" }, "dependencies": { - "@budibase/bbui": "^1.0.27-alpha.6", + "@budibase/bbui": "^1.0.27-alpha.17", "@budibase/standard-components": "^0.9.139", - "@budibase/string-templates": "^1.0.27-alpha.6", + "@budibase/string-templates": "^1.0.27-alpha.17", "regexparam": "^1.3.0", "rollup-plugin-polyfill-node": "^0.8.0", "shortid": "^2.2.15", diff --git a/packages/client/src/api/datasources.js b/packages/client/src/api/datasources.js deleted file mode 100644 index e5e9bad7d..000000000 --- a/packages/client/src/api/datasources.js +++ /dev/null @@ -1,145 +0,0 @@ -import { cloneDeep } from "lodash/fp" -import { fetchTableData, fetchTableDefinition } from "./tables" -import { fetchViewData } from "./views" -import { fetchRelationshipData } from "./relationships" -import { FieldTypes } from "../constants" -import { executeQuery, fetchQueryDefinition } from "./queries" -import { - convertJSONSchemaToTableSchema, - getJSONArrayDatasourceSchema, -} from "builder/src/builderStore/jsonUtils" -import API from "./api.js" - -/** - * Fetches all rows for a particular Budibase data source. - */ -export const fetchDatasource = async dataSource => { - if (!dataSource || !dataSource.type) { - return [] - } - - // Fetch all rows in data source - const { type, tableId, fieldName } = dataSource - let rows = [], - info = {} - if (type === "table") { - rows = await fetchTableData(tableId) - } else if (type === "view") { - rows = await fetchViewData(dataSource) - } else if (type === "query") { - // Set the default query params - let parameters = cloneDeep(dataSource.queryParams || {}) - for (let param of dataSource.parameters) { - if (!parameters[param.name]) { - parameters[param.name] = param.default - } - } - const { data, ...rest } = await executeQuery({ - queryId: dataSource._id, - parameters, - }) - info = rest - rows = data - } else if (type === FieldTypes.LINK) { - rows = await fetchRelationshipData({ - rowId: dataSource.rowId, - tableId: dataSource.rowTableId, - fieldName, - }) - } - - // Enrich the result is always an array - return { rows: Array.isArray(rows) ? rows : [], info } -} - -/** - * Fetches the schema of any kind of datasource. - */ -export const fetchDatasourceSchema = async dataSource => { - if (!dataSource) { - return null - } - const { type } = dataSource - let schema - - // Nested providers should already have exposed their own schema - if (type === "provider") { - schema = dataSource.value?.schema - } - - // Field sources have their schema statically defined - if (type === "field") { - if (dataSource.fieldType === "attachment") { - schema = { - url: { - type: "string", - }, - name: { - type: "string", - }, - } - } else if (dataSource.fieldType === "array") { - schema = { - value: { - type: "string", - }, - } - } - } - - // JSON arrays need their table definitions fetched. - // We can then extract their schema as a subset of the table schema. - if (type === "jsonarray") { - const table = await fetchTableDefinition(dataSource.tableId) - schema = getJSONArrayDatasourceSchema(table?.schema, dataSource) - } - - // Tables, views and links can be fetched by table ID - if ( - (type === "table" || type === "view" || type === "link") && - dataSource.tableId - ) { - const table = await fetchTableDefinition(dataSource.tableId) - schema = table?.schema - } - - // Queries can be fetched by query ID - if (type === "query" && dataSource._id) { - const definition = await fetchQueryDefinition(dataSource._id) - schema = definition?.schema - } - - // Sanity check - if (!schema) { - return null - } - - // Check for any JSON fields so we can add any top level properties - let jsonAdditions = {} - Object.keys(schema).forEach(fieldKey => { - const fieldSchema = schema[fieldKey] - if (fieldSchema?.type === "json") { - const jsonSchema = convertJSONSchemaToTableSchema(fieldSchema, { - squashObjects: true, - }) - Object.keys(jsonSchema).forEach(jsonKey => { - jsonAdditions[`${fieldKey}.${jsonKey}`] = { - type: jsonSchema[jsonKey].type, - nestedJSON: true, - } - }) - } - }) - return { ...schema, ...jsonAdditions } -} - -export const getSignedS3URL = async (datasourceId, bucket, key) => { - if (!datasourceId) { - return null - } - const res = await API.post({ - url: `/api/datasources/s3/getSignedS3URL`, - body: { datasourceId, bucket, key }, - }) - return res?.signedUrl -} diff --git a/packages/client/src/api/index.js b/packages/client/src/api/index.js index 1fd5b1813..d429eb437 100644 --- a/packages/client/src/api/index.js +++ b/packages/client/src/api/index.js @@ -1,6 +1,5 @@ export * from "./rows" export * from "./auth" -export * from "./datasources" export * from "./tables" export * from "./attachments" export * from "./views" diff --git a/packages/client/src/api/queries.js b/packages/client/src/api/queries.js index 834282fe7..e8972f657 100644 --- a/packages/client/src/api/queries.js +++ b/packages/client/src/api/queries.js @@ -4,7 +4,7 @@ import API from "./api" /** * Executes a query against an external data connector. */ -export const executeQuery = async ({ queryId, parameters }) => { +export const executeQuery = async ({ queryId, pagination, parameters }) => { const query = await fetchQueryDefinition(queryId) if (query?.datasourceId == null) { notificationStore.actions.error("That query couldn't be found") @@ -14,6 +14,7 @@ export const executeQuery = async ({ queryId, parameters }) => { url: `/api/v2/queries/${queryId}`, body: { parameters, + pagination, }, }) if (res.error) { diff --git a/packages/client/src/components/Component.svelte b/packages/client/src/components/Component.svelte index d8cdbd3f5..d05f01f58 100644 --- a/packages/client/src/components/Component.svelte +++ b/packages/client/src/components/Component.svelte @@ -19,6 +19,16 @@ export let isScreen = false export let isBlock = false + // Ref to the svelte component + let ref + + // Initial settings are passed in on first render of the component. + // When the first instance of cachedSettings are set, this object is set to + // reference cachedSettings, so that mutations to cachedSettings also affect + // initialSettings, but it does not get caught by svelte invalidation - which + // would happen if we spread cachedSettings directly to the component. + let initialSettings + // Component settings are the un-enriched settings for this component that // need to be enriched at this level. // Nested settings are the un-enriched block settings that are to be passed on @@ -267,16 +277,26 @@ const cacheSettings = (enriched, nested, conditional) => { const allSettings = { ...enriched, ...nested, ...conditional } if (!cachedSettings) { - cachedSettings = allSettings + cachedSettings = { ...allSettings } + initialSettings = cachedSettings } else { Object.keys(allSettings).forEach(key => { - if (!propsAreSame(allSettings[key], cachedSettings[key])) { + const same = propsAreSame(allSettings[key], cachedSettings[key]) + if (!same) { cachedSettings[key] = allSettings[key] + assignSetting(key, allSettings[key]) } }) } } + // Assigns a certain setting to this component. + // We manually use the svelte $set function to avoid triggering additional + // reactive statements. + const assignSetting = (key, value) => { + ref?.$$set?.({ [key]: value }) + } + // Generates a key used to determine when components need to fully remount. // Currently only toggling editing requires remounting. const getRenderKey = (id, editing) => { @@ -299,7 +319,7 @@ data-id={id} data-name={name} > - + {#if children.length} {#each children as child (child._id)} diff --git a/packages/client/src/components/app/DataProvider.svelte b/packages/client/src/components/app/DataProvider.svelte index ec328c60c..71c54db4d 100644 --- a/packages/client/src/components/app/DataProvider.svelte +++ b/packages/client/src/components/app/DataProvider.svelte @@ -1,13 +1,9 @@
- {#if !loaded} + {#if !$fetch.loaded}
@@ -328,14 +134,14 @@ {:else} {/if} - {#if paginate && internalTable} + {#if paginate && $fetch.supportsPagination} {/if} diff --git a/packages/client/src/components/app/blocks/CardsBlock.svelte b/packages/client/src/components/app/blocks/CardsBlock.svelte index 3a136b5b2..301b440ab 100644 --- a/packages/client/src/components/app/blocks/CardsBlock.svelte +++ b/packages/client/src/components/app/blocks/CardsBlock.svelte @@ -30,7 +30,7 @@ export let cardButtonOnClick export let linkColumn - const { API, styleable } = getContext("sdk") + const { fetchDatasourceSchema, styleable } = getContext("sdk") const context = getContext("context") const component = getContext("component") const schemaComponentMap = { @@ -45,6 +45,7 @@ let dataProviderId let repeaterId let schema + let schemaLoaded = false $: fetchSchema(dataSource) $: enrichedSearchColumns = enrichSearchColumns(searchColumns, schema) @@ -111,103 +112,106 @@ // Load the datasource schema so we can determine column types const fetchSchema = async dataSource => { if (dataSource) { - schema = await API.fetchDatasourceSchema(dataSource) + schema = await fetchDatasourceSchema(dataSource) } + schemaLoaded = true } - -
- - {#if title || enrichedSearchColumns?.length || showTitleButton} -
-
- {title || ""} +{#if schemaLoaded} + +
+ + {#if title || enrichedSearchColumns?.length || showTitleButton} +
+
+ {title || ""} +
+
+ {#if enrichedSearchColumns?.length} +
+ {#each enrichedSearchColumns as column} + + {/each} +
+ {/if} + {#if showTitleButton} + + {/if} +
-
- {#if enrichedSearchColumns?.length} -
- {#each enrichedSearchColumns as column} - - {/each} -
- {/if} - {#if showTitleButton} - - {/if} -
-
- {/if} - + {/if} + > + + - -
- +
+ +{/if}