Skip to content

Locally failing tests.  #60

Open
Open
Locally failing tests. #60

Description

@saulshanabrook
saulshanabrook
opened on Mar 24, 2020

When I run the tests locally, most of them fail. Also a dialogue pops up to ask if I want to save one of the notebooks

Most of them fail I believe because the images dont match. I looked at the files, and all of the look the same, but maybe it thinks they look different. Maybe comparing PNGs would be better?

Screen Shot 2020-03-23 at 8 42 06 PM

yarn run test
yarn run v1.19.2
$ env JEST_PUPPETEER_CONFIG=./config/jest/jest-puppeteer.config.js jest --runInBand --verbose=true --detectOpenHandles --config ./config/jest/jest.config.js
Determining test suites to run...[W 20:39:54.567 LabApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 20:39:55.902 LabApp] JupyterLab extension loaded from /usr/local/Caskroom/miniconda/base/envs/ibis-vega-transform/lib/python3.7/site-packages/jupyterlab
[I 20:39:55.903 LabApp] JupyterLab application directory is /usr/local/Caskroom/miniconda/base/envs/ibis-vega-transform/share/jupyter/lab
[I 20:39:55.906 LabApp] Serving notebooks from local directory: /Users/saul/p/ibis-vega-transform
[I 20:39:55.906 LabApp] The Jupyter Notebook is running at:
[I 20:39:55.906 LabApp] http://localhost:8080/
[I 20:39:55.906 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
  console.log test/ui/test.ts:229


    ## Running "charting-example.ipynb"


[W 20:40:00.358 LabApp] Notebook examples/performance-charts.ipynb is not trusted
[W 20:40:00.365 LabApp] Notebook examples/charting-example.ipynb is not trusted
INFO:     Started server process [92232]
INFO:     Uvicorn running on http://0.0.0.0:59491 (Press CTRL+C to quit)
INFO:     Waiting for application startup.
INFO:     Application startup complete.
[I 20:40:00.720 LabApp] Kernel started: e3a651da-b07e-40ad-8492-d657a14f8a2e
[I 20:40:00.734 LabApp] Kernel started: b21450b8-bfe1-495f-91f7-5919fd79a706
[IPKernelApp] ERROR | No such comm target registered: ibis-vega-transform:compiler
[IPKernelApp] ERROR | No such comm target registered: ibis-vega-transform:compiler
[I 20:40:08.438 LabApp] Build is up to date
[I 20:40:08.854 LabApp] Kernel restarted: b21450b8-bfe1-495f-91f7-5919fd79a706
INFO:     Reporting span 30c04757b7533a6e:41ceea03b33579ca:ffded251c796b7a8:1 browser.compileSpec
INFO:     Reporting span 30c04757b7533a6e:3fe2d87804cac78:ffded251c796b7a8:1 browser.vegaEmbed
INFO:     Reporting span 30c04757b7533a6e:ffded251c796b7a8:a2995eeccea36028:1 browser.renderModel
  console.log test/ui/test.ts:253
    Found charts: 1

INFO:     Reporting span 30c04757b7533a6e:ac95546e57be8303:a2995eeccea36028:1 browser.transform
INFO:     Reporting span 30c04757b7533a6e:3d6ef33d2f01d6e2:a2995eeccea36028:1 browser.transform
INFO:     Reporting span 30c04757b7533a6e:475f0e403b0e79be:a2995eeccea36028:1 browser.transform
INFO:     Reporting span 30c04757b7533a6e:429e0ee91115ca54:a2995eeccea36028:1 browser.transform
INFO:     Reporting span 30c04757b7533a6e:5d2c5875c3121846:a2995eeccea36028:1 browser.transform
INFO:     Reporting span 30c04757b7533a6e:56c7a28706fef1a4:a2995eeccea36028:1 browser.transform
  console.log test/ui/test.ts:261
    Setting download folder: /Users/saul/p/images/charting-example

  console.log test/ui/test.ts:164
    File saved at: /Users/saul/p/images/charting-example/visualization.svg

  console.log test/ui/test.ts:165
    File renamed to: /Users/saul/p/images/charting-example/0.svg

  console.log test/ui/test.ts:280
    Original images folder: /Users/saul/p/ibis-vega-transform/images/charting-example

  console.log test/ui/test.ts:199
    Comparing images: "0.svg"

  console.log test/ui/test.ts:200
    /Users/saul/p/ibis-vega-transform/images/charting-example/0.svg

  console.log test/ui/test.ts:201
    /Users/saul/p/images/charting-example/0.svg

  console.log test/ui/test.ts:209
    {
      isSameDimensions: true,
      dimensionDifference: { width: 0, height: 0 },
      rawMisMatchPercentage: 1.1513410108568976,
      misMatchPercentage: '1.15',
      diffBounds: { top: 666, left: 237, bottom: 949, right: 922 },
      analysisTime: 1261,
      getImageDataUrl: [Function]
    }

  console.log test/ui/test.ts:229


    ## Running "ibis-altair-extraction.ipynb"


  console.log test/ui/test.ts:229


    ## Running "interactive-slider.ipynb"


  console.log test/ui/test.ts:229


    ## Running "omnisci-vega-example.ipynb"


[I 20:42:07.896 LabApp] Starting buffering for e3a651da-b07e-40ad-8492-d657a14f8a2e:1ffa6aa6-870c-4ad6-91a1-bd4b63ca6790
[I 20:42:07.897 LabApp] Starting buffering for b21450b8-bfe1-495f-91f7-5919fd79a706:ab568c93-8ab2-4161-a4e7-2ee96a01a950
[W 20:42:09.553 LabApp] Notebook examples/charting-example.ipynb is not trusted
[W 20:42:09.565 LabApp] Notebook examples/performance-charts.ipynb is not trusted
[W 20:42:10.148 LabApp] Notebook examples/omnisci-vega-example.ipynb is not trusted
[I 20:42:10.359 LabApp] Kernel started: 82351599-f707-4a25-bb01-3791c110967c
[I 20:42:17.893 LabApp] Kernel restarted: 82351599-f707-4a25-bb01-3791c110967c
[I 20:42:19.110 LabApp] Build is up to date
INFO:     Reporting span 76a0faac88c3c1d1:28af9d419b23d430:e993a7e02ef673c5:1 browser.compileSpec
INFO:     Reporting span 76a0faac88c3c1d1:b202b2383344cceb:e993a7e02ef673c5:1 browser.vegaEmbed
INFO:     Reporting span 76a0faac88c3c1d1:e993a7e02ef673c5:1a644b09f479e532:1 browser.renderModel
  console.log test/ui/test.ts:253
    Found charts: 1

  console.log test/ui/test.ts:261
    Setting download folder: /Users/saul/p/images/omnisci-vega-example

INFO:     Reporting span 76a0faac88c3c1d1:26cc93b71d8ab551:1a644b09f479e532:1 browser.transform
  console.log test/ui/test.ts:164
    File saved at: /Users/saul/p/images/omnisci-vega-example/visualization.svg

  console.log test/ui/test.ts:165
    File renamed to: /Users/saul/p/images/omnisci-vega-example/0.svg

  console.log test/ui/test.ts:280
    Original images folder: /Users/saul/p/ibis-vega-transform/images/omnisci-vega-example

  console.log test/ui/test.ts:199
    Comparing images: "0.svg"

  console.log test/ui/test.ts:200
    /Users/saul/p/ibis-vega-transform/images/omnisci-vega-example/0.svg

  console.log test/ui/test.ts:201
    /Users/saul/p/images/omnisci-vega-example/0.svg

  console.log test/ui/test.ts:209
    {
      isSameDimensions: true,
      dimensionDifference: { width: 0, height: 0 },
      rawMisMatchPercentage: 0,
      misMatchPercentage: '0.00',
      diffBounds: { top: 606, left: 476, bottom: 0, right: 0 },
      analysisTime: 348,
      getImageDataUrl: [Function]
    }

[I 20:42:50.553 LabApp] Kernel shutdown: 82351599-f707-4a25-bb01-3791c110967c
[I 20:42:54.587 LabApp] Starting buffering for b21450b8-bfe1-495f-91f7-5919fd79a706:3373411d-5447-4852-a2be-571135a38c78
[I 20:42:54.587 LabApp] Starting buffering for e3a651da-b07e-40ad-8492-d657a14f8a2e:e5ddb438-84f3-41f4-a245-fac2e68a953f
  console.log test/ui/test.ts:229


    ## Running "performance-charts.ipynb"


[W 20:42:55.700 LabApp] Notebook examples/performance-charts.ipynb is not trusted
[W 20:42:55.709 LabApp] Notebook examples/charting-example.ipynb is not trusted
[I 20:43:04.085 LabApp] Kernel restarted: e3a651da-b07e-40ad-8492-d657a14f8a2e
[I 20:43:05.152 LabApp] Build is up to date
INFO:     Reporting span 91f23b4b96ed2827:46a15660395bfac:68dbe4c7d2c69d27:1 browser.compileSpec
INFO:     Reporting span cc63502bd76ae0f0:f3f14f0c462254fe:aa4291b947ecaf5e:1 browser.compileSpec
INFO:     Reporting span 91f23b4b96ed2827:7cb585f01f74437e:68dbe4c7d2c69d27:1 browser.vegaEmbed
INFO:     Reporting span cc63502bd76ae0f0:c211aa6adc411f36:aa4291b947ecaf5e:1 browser.vegaEmbed
INFO:     Reporting span 91f23b4b96ed2827:68dbe4c7d2c69d27:68219de9377e5bfb:1 browser.renderModel
INFO:     Reporting span cc63502bd76ae0f0:aa4291b947ecaf5e:c827d875909afb22:1 browser.renderModel
  console.log test/ui/test.ts:253
    Found charts: 2

INFO:     Reporting span 91f23b4b96ed2827:b747a24778ab9aa4:68219de9377e5bfb:1 browser.transform
INFO:     Reporting span 91f23b4b96ed2827:206f6c66071ee26:68219de9377e5bfb:1 browser.transform
INFO:     Reporting span 91f23b4b96ed2827:dbf10a1d59799b6f:68219de9377e5bfb:1 browser.transform
INFO:     Reporting span 91f23b4b96ed2827:454f1fa0c38198d0:68219de9377e5bfb:1 browser.transform
INFO:     Reporting span 91f23b4b96ed2827:da2006d55cc4c869:68219de9377e5bfb:1 browser.transform
INFO:     Reporting span 91f23b4b96ed2827:59ae9dcd17bf109f:68219de9377e5bfb:1 browser.transform
  console.log test/ui/test.ts:261
    Setting download folder: /Users/saul/p/images/performance-charts

INFO:     Reporting span cc63502bd76ae0f0:9845549a4195e069:c827d875909afb22:1 browser.transform
  console.log test/ui/test.ts:164
    File saved at: /Users/saul/p/images/performance-charts/visualization.svg

  console.log test/ui/test.ts:165
    File renamed to: /Users/saul/p/images/performance-charts/0.svg

  console.log test/ui/test.ts:164
    File saved at: /Users/saul/p/images/performance-charts/visualization.svg

  console.log test/ui/test.ts:165
    File renamed to: /Users/saul/p/images/performance-charts/1.svg

  console.log test/ui/test.ts:280
    Original images folder: /Users/saul/p/ibis-vega-transform/images/performance-charts

  console.log test/ui/test.ts:199
    Comparing images: "0.svg"

  console.log test/ui/test.ts:200
    /Users/saul/p/ibis-vega-transform/images/performance-charts/0.svg

  console.log test/ui/test.ts:201
    /Users/saul/p/images/performance-charts/0.svg

  console.log test/ui/test.ts:209
    {
      isSameDimensions: true,
      dimensionDifference: { width: 0, height: 0 },
      rawMisMatchPercentage: 1.1513410108568976,
      misMatchPercentage: '1.15',
      diffBounds: { top: 666, left: 237, bottom: 949, right: 922 },
      analysisTime: 1301,
      getImageDataUrl: [Function]
    }

  console.log test/ui/test.ts:199
    Comparing images: "1.svg"

  console.log test/ui/test.ts:200
    /Users/saul/p/ibis-vega-transform/images/performance-charts/1.svg

  console.log test/ui/test.ts:201
    /Users/saul/p/images/performance-charts/1.svg

  console.log test/ui/test.ts:209
    {
      isSameDimensions: true,
      dimensionDifference: { width: 0, height: 0 },
      rawMisMatchPercentage: 0.22698427504953797,
      misMatchPercentage: '0.23',
      diffBounds: { top: 48, left: 127, bottom: 509, right: 462 },
      analysisTime: 426,
      getImageDataUrl: [Function]
    }

  console.log test/ui/test.ts:229


    ## Running "vega-compiler.ipynb"


  console.log test/ui/test.ts:292


[I 20:44:10.322 LabApp] Starting buffering for b21450b8-bfe1-495f-91f7-5919fd79a706:fd9d58ed-50e6-4f3d-805f-2d3282af3094
[I 20:44:10.323 LabApp] Starting buffering for e3a651da-b07e-40ad-8492-d657a14f8a2e:f131b4bc-83ae-4229-bb33-7fbf449c30b2
 FAIL  test/ui/test.ts (251.952s)
  Test Ibis-Vega-Transform
    ✕ should open and run notebook correctly (62416ms)
    ✕ should open and run notebook correctly (30005ms)
    ✕ should open and run notebook correctly (30008ms)
    ✓ should open and run notebook correctly (55116ms)
    ✕ should open and run notebook correctly (43723ms)
    ✕ should open and run notebook correctly (30003ms)

  ● Test Ibis-Vega-Transform › should open and run notebook correctly

    expect(received).toBe(expected) // Object.is equality

    Expected: true
    Received: false

      285 |       );
      286 |       const isBelowThreshold = value => value < threshold;
    > 287 |       expect(results.every(isBelowThreshold)).toBe(true);
          |                                               ^
      288 |       await page.waitFor(2 * 1000);
      289 |
      290 |       // Close dialog "Are you sure you want to leave..."

      at test/ui/test.ts:287:47

  ● Test Ibis-Vega-Transform › should open and run notebook correctly

    TimeoutError: Navigation timeout of 30000 ms exceeded

      230 |
      231 |       // Go to specific notebook file
    > 232 |       await page.goto(`http://localhost:8080/lab/tree/examples/${file}.ipynb`);
          |       ^
      233 |       await page.waitForSelector(
      234 |         '.lm-DockPanel-tabBar li[data-type="document-title"]'
      235 |       );

      at node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
      at FrameManager.navigateFrame (node_modules/puppeteer/lib/FrameManager.js:94:17)
      at Frame.goto (node_modules/puppeteer/lib/FrameManager.js:406:12)
      at Page.goto (node_modules/puppeteer/lib/Page.js:672:12)
      at test/ui/test.ts:232:7
        -- ASYNC --
      at Frame.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
      at Page.goto (node_modules/puppeteer/lib/Page.js:672:49)
      at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:112:23)
      at test/ui/test.ts:232:18
        -- ASYNC --
      at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
      at test/ui/test.ts:232:18

  ● Test Ibis-Vega-Transform › should open and run notebook correctly

    TimeoutError: Navigation timeout of 30000 ms exceeded

      230 |
      231 |       // Go to specific notebook file
    > 232 |       await page.goto(`http://localhost:8080/lab/tree/examples/${file}.ipynb`);
          |       ^
      233 |       await page.waitForSelector(
      234 |         '.lm-DockPanel-tabBar li[data-type="document-title"]'
      235 |       );

      at node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
      at FrameManager.navigateFrame (node_modules/puppeteer/lib/FrameManager.js:94:17)
      at Frame.goto (node_modules/puppeteer/lib/FrameManager.js:406:12)
      at Page.goto (node_modules/puppeteer/lib/Page.js:672:12)
      at test/ui/test.ts:232:7
        -- ASYNC --
      at Frame.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
      at Page.goto (node_modules/puppeteer/lib/Page.js:672:49)
      at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:112:23)
      at test/ui/test.ts:232:18
        -- ASYNC --
      at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
      at test/ui/test.ts:232:18

  ● Test Ibis-Vega-Transform › should open and run notebook correctly

    expect(received).toBe(expected) // Object.is equality

    Expected: true
    Received: false

      285 |       );
      286 |       const isBelowThreshold = value => value < threshold;
    > 287 |       expect(results.every(isBelowThreshold)).toBe(true);
          |                                               ^
      288 |       await page.waitFor(2 * 1000);
      289 |
      290 |       // Close dialog "Are you sure you want to leave..."

      at test/ui/test.ts:287:47

  ● Test Ibis-Vega-Transform › should open and run notebook correctly

    TimeoutError: Navigation timeout of 30000 ms exceeded

      230 |
      231 |       // Go to specific notebook file
    > 232 |       await page.goto(`http://localhost:8080/lab/tree/examples/${file}.ipynb`);
          |                  ^
      233 |       await page.waitForSelector(
      234 |         '.lm-DockPanel-tabBar li[data-type="document-title"]'
      235 |       );

      at node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
        -- ASYNC --
      at Frame.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
      at Page.goto (node_modules/puppeteer/lib/Page.js:672:49)
      at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:112:23)
      at test/ui/test.ts:232:18
        -- ASYNC --
      at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
      at test/ui/test.ts:232:18

Test Suites: 1 failed, 1 total
Tests:       5 failed, 1 passed, 6 total
Snapshots:   0 total
Time:        254.296s, estimated 305s
Ran all test suites.
INFO:     Shutting down
[I 20:44:10.519 LabApp] Shutting down 2 kernels
INFO:     Finished server process [92232]
[I 20:44:10.624 LabApp] Kernel shutdown: e3a651da-b07e-40ad-8492-d657a14f8a2e
[I 20:44:10.625 LabApp] Kernel shutdown: b21450b8-bfe1-495f-91f7-5919fd79a706
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions