Skip to content

initial changes to implement fog cloud storage library #822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
icarito wants to merge 11 commits into
base: main
Choose a base branch
from
Open

initial changes to implement fog cloud storage library #822

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
662c128
initial changes to implement fog cloud storage library
icarito Dec 28, 2021
017b94b
gemfile lock
jywarren Jan 4, 2022
430808f
Update config/initializers/paperclip_string.rb
jywarren Jan 4, 2022
74bb572
photo.copy_to_local_file(:original) before local ImageMagick - https:…
jywarren Jan 25, 2022
ada364c
initial changes to implement fog cloud storage library
icarito Dec 28, 2021
9735504
Update config/initializers/paperclip_string.rb
jywarren Jan 4, 2022
16d595c
,local_photo_path added to copy_to_local_file()
jywarren Jan 25, 2022
eeb3446
remove old test
jywarren Jan 25, 2022
37da66e
save to tmp
jywarren Jan 25, 2022
8bf0286
add: image.write(local_photo_path)
jywarren Feb 8, 2022
3e33b58
Avoid resetting database on rebuild
icarito Feb 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ gem 'tzinfo', '~> 1.2'
gem 'open_id_authentication'
gem 'protected_attributes_continued', '~> 1.8.2'
gem 'ruby-openid'
gem 'fog-google'
Copy link
Copy Markdown
Member

@jywarren jywarren Jan 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do this due to fog/fog-google#535

Suggested change
gem 'fog-google'
gem 'fog-google', '1.13.0' # Limited to `v1.13.0` due to https://github.com/fog/fog-google/issues/535

gem 'fog-local'

gem 'rack-offline', '>= 0.6.4'

Expand Down
105 changes: 105 additions & 0 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,23 +62,108 @@ GEM
climate_control (0.2.0)
concurrent-ruby (1.1.9)
crass (1.0.6)
declarative (0.0.20)
docile (1.4.0)
erubi (1.10.0)
excon (0.90.0)
execjs (2.7.0)
faker (2.19.0)
i18n (>= 1.6, < 2)
faraday (1.9.3)
faraday-em_http (~> 1.0)
faraday-em_synchrony (~> 1.0)
faraday-excon (~> 1.1)
faraday-httpclient (~> 1.0)
faraday-multipart (~> 1.0)
faraday-net_http (~> 1.0)
faraday-net_http_persistent (~> 1.0)
faraday-patron (~> 1.0)
faraday-rack (~> 1.0)
faraday-retry (~> 1.0)
ruby2_keywords (>= 0.0.4)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
multipart-post (>= 1.2, < 3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
ffi (1.15.5)
fog-core (2.1.0)
builder
excon (~> 0.58)
formatador (~> 0.2)
mime-types
fog-google (1.17.0)
fog-core (<= 2.1.0)
fog-json (~> 1.2)
fog-xml (~> 0.1.0)
google-apis-compute_v1 (~> 0.14)
google-apis-dns_v1 (~> 0.12)
google-apis-iamcredentials_v1 (~> 0.6)
google-apis-monitoring_v3 (~> 0.12)
google-apis-pubsub_v1 (~> 0.7)
google-apis-sqladmin_v1beta4 (~> 0.13)
google-apis-storage_v1 (~> 0.6)
google-cloud-env (~> 1.2)
fog-json (1.2.0)
fog-core
multi_json (~> 1.10)
fog-local (0.8.0)
fog-core (>= 1.27, < 3.0)
fog-xml (0.1.4)
fog-core
nokogiri (>= 1.5.11, < 2.0.0)
font-awesome-rails (4.7.0.8)
railties (>= 3.2, < 8.0)
formatador (0.3.0)
globalid (0.4.2)
activesupport (>= 4.2.0)
google-apis-compute_v1 (0.23.0)
google-apis-core (>= 0.4, < 2.a)
google-apis-core (0.4.2)
addressable (~> 2.5, >= 2.5.1)
googleauth (>= 0.16.2, < 2.a)
httpclient (>= 2.8.1, < 3.a)
mini_mime (~> 1.0)
representable (~> 3.0)
retriable (>= 2.0, < 4.a)
rexml
webrick
google-apis-dns_v1 (0.18.0)
google-apis-core (>= 0.4, < 2.a)
google-apis-iamcredentials_v1 (0.10.0)
google-apis-core (>= 0.4, < 2.a)
google-apis-monitoring_v3 (0.20.0)
google-apis-core (>= 0.4, < 2.a)
google-apis-pubsub_v1 (0.12.0)
google-apis-core (>= 0.4, < 2.a)
google-apis-sqladmin_v1beta4 (0.22.0)
google-apis-core (>= 0.4, < 2.a)
google-apis-storage_v1 (0.11.0)
google-apis-core (>= 0.4, < 2.a)
google-cloud-env (1.5.0)
faraday (>= 0.17.3, < 2.0)
googleauth (1.1.0)
faraday (>= 0.17.3, < 2.0)
jwt (>= 1.4, < 3.0)
memoist (~> 0.16)
multi_json (~> 1.11)
os (>= 0.9, < 2.0)
signet (>= 0.16, < 2.a)
httpclient (2.8.3)
i18n (1.8.11)
concurrent-ruby (~> 1.0)
jquery-rails (4.4.0)
rails-dom-testing (>= 1, < 3)
railties (>= 4.2.0)
thor (>= 0.14, < 2.0)
json (2.5.1)
jwt (2.3.0)
libv8 (3.16.14.19)
listen (3.7.1)
rb-fsevent (~> 0.10, >= 0.10.3)
Expand All @@ -91,6 +176,7 @@ GEM
marcel (0.3.3)
mimemagic (~> 0.3.2)
matrix (0.4.2)
memoist (0.16.2)
method_source (1.0.0)
mime-types (3.4.1)
mime-types-data (~> 3.2015)
Expand All @@ -102,13 +188,16 @@ GEM
mini_portile2 (2.6.1)
minitest (5.15.0)
msgpack (1.4.2)
multi_json (1.15.0)
multipart-post (2.1.1)
mysql2 (0.5.3)
nio4r (2.5.8)
nokogiri (1.12.5)
mini_portile2 (~> 2.6.1)
racc (~> 1.4)
open_id_authentication (1.3.0)
rack-openid (~> 1.3)
os (1.1.4)
paperclip (6.1.0)
activemodel (>= 4.2.0)
activesupport (>= 4.2.0)
Expand Down Expand Up @@ -176,9 +265,14 @@ GEM
redcarpet (3.5.1)
ref (2.0.0)
regexp_parser (2.2.0)
representable (3.1.1)
declarative (< 0.1.0)
trailblazer-option (>= 0.1.1, < 0.2.0)
uber (< 0.2.0)
responders (3.0.1)
actionpack (>= 5.0)
railties (>= 5.0)
retriable (3.1.2)
rexml (3.2.5)
rmagick (4.2.4)
rubocop (1.25.0)
Expand All @@ -197,10 +291,16 @@ GEM
rubocop-ast (>= 0.4.0)
ruby-openid (2.9.2)
ruby-progressbar (1.11.0)
ruby2_keywords (0.0.5)
rubyzip (2.3.2)
selenium-webdriver (3.142.7)
childprocess (>= 0.5, < 4.0)
rubyzip (>= 1.2.2)
signet (0.16.0)
addressable (~> 2.8)
faraday (>= 0.17.3, < 2.0)
jwt (>= 1.5, < 3.0)
multi_json (~> 1.10)
simplecov (0.21.2)
docile (~> 1.1)
simplecov-html (~> 0.11)
Expand All @@ -223,11 +323,14 @@ GEM
ref
thor (1.1.0)
thread_safe (0.3.6)
trailblazer-option (0.1.2)
tzinfo (1.2.9)
thread_safe (~> 0.1)
uber (0.1.0)
uglifier (4.2.0)
execjs (>= 0.3.0, < 3)
unicode-display_width (2.1.0)
webrick (1.7.0)
websocket-driver (0.7.3)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
Expand All @@ -244,6 +347,8 @@ DEPENDENCIES
bootsnap (~> 1.10.1)
capybara
faker (~> 2.19.0)
fog-google
fog-local
font-awesome-rails
jquery-rails
listen (~> 3.7.1)
Expand Down
7 changes: 0 additions & 7 deletions app/assets/javascripts/analyze.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,13 +290,6 @@ $W = {
f.submit();
},

set_sample_row: function() {
var rownum = prompt('Enter the percentage from the top edge from which you would like to extract a spectrum.','100')
$("#imagelink")[0].onclick = ""
$('#imagelink').tooltip('destroy')
if (rownum) window.location = '/spectrums/setsamplerow/'+$W.spectrum_id+'?row='+rownum/100.00
},

click_to_set_sample_row: function() {
$("#imagelink")[0].onclick = ""
$('#imagelink').tooltip('destroy')
Expand Down
15 changes: 0 additions & 15 deletions app/controllers/spectrums_controller.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,21 +468,6 @@ def match
render html: @spectrum.find_match_in_set(params[:set]).to_json
end

# Start doing this client side!
def setsamplerow
require 'rubygems'
require 'rmagick'
@spectrum = Spectrum.find params[:id]
require_ownership(@spectrum)
image = Magick::ImageList.new('public' + (@spectrum.photo.url.split('?')[0]).gsub('%20', ' '))
@spectrum.sample_row = (params[:row].to_f * image.rows)
@spectrum.extract_data
@spectrum.save
flash[:warning] = "If this spectrum image is not perfectly vertical, you may need to recalibrate after <a href='//publiclab.org/wiki/spectral-workbench-calibration#Cross+section'>setting a new cross-section</a>."
redirect_to spectrum_path(@spectrum)
end

# Start doing this client side!
def find_brightest_row
@spectrum = Spectrum.find params[:id]
require_ownership(@spectrum)
Expand Down
19 changes: 14 additions & 5 deletions app/models/spectrum.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,8 @@ def self.weekly_tallies

# finds the brightest row of the image and uses that as its sample row
def find_brightest_row
image = Magick::ImageList.new('public' + (photo.url.split('?')[0]).gsub('%20', ' '))
photo.copy_to_local_file(:original,local_photo_path)
Copy link
Copy Markdown
Member

@jywarren jywarren Jan 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://rubydoc.info/gems/paperclip/Paperclip/Storage/S3#copy_to_local_file-instance_method

image = Magick::ImageList.new(local_photo_path)
brightest_row = 0
brightest = 0
# sum brightness for each row
Expand All @@ -140,7 +141,8 @@ def find_brightest_row
def extract_data
pixels = []

image = Magick::ImageList.new('public' + (photo.url.split('?')[0]).gsub('%20', ' '))
photo.copy_to_local_file(:original,local_photo_path)
image = Magick::ImageList.new(local_photo_path)
# saved sample_row may be greater than image height, so temporarily compensate,
# but preserve sample_row in case we rotate back or something
self.sample_row = image.rows - 2 if sample_row > image.rows
Expand Down Expand Up @@ -278,15 +280,17 @@ def clone_calibration(clone_id)

# rotate clockwise
def rotate
image = Magick::ImageList.new('public' + (photo.url.split('?')[0]).gsub('%20', ' '))
photo.copy_to_local_file(:original,local_photo_path)
image = Magick::ImageList.new(local_photo_path)
image.rotate!(-90)
image.write('public' + photo.url)
photo.reprocess!
end

# horizontally flips image to match reversed spectrum, toggles 'reversed' flag
def reverse
image = Magick::ImageList.new('public' + (photo.url.split('?')[0]).gsub('%20', ' '))
photo.copy_to_local_file(:original,local_photo_path)
image = Magick::ImageList.new(local_photo_path)
image.flop!
image.write('public' + photo.url)
self.reversed = !reversed
Expand Down Expand Up @@ -629,4 +633,9 @@ def find_similar(range)
def spectrum_params
params.require(:spectrum).permit(:title, :author, :user_id, :notes, :photo, :video_row, :data)
end
end

def local_photo_path
'public' + (photo.url.split('?')[0]).gsub('%20', ' ')
end

end
11 changes: 11 additions & 0 deletions config/initializers/paperclip_string.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@
# Paperclip.options[:command_path] = "/usr/bin/identify"
# Paperclip.options[:command_path] = "/usr/local/bin"

Paperclip::Attachment.default_options[:storage] = :fog
Paperclip::Attachment.default_options[:fog_directory] = ENV["GOOGLE_STORAGE_BUCKET_NAME"] || ''
Paperclip::Attachment.default_options[:path] = ":rails_root/public/system/:attachment/:id/:style/:filename"
Copy link
Copy Markdown
Member

@jywarren jywarren Jan 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Paperclip::Attachment.default_options[:path] = ":rails_root/public/system/:attachment/:id/:style/:filename"
Paperclip::Attachment.default_options[:path] = ":rails_root/public/system/public/system/:attachment/:id/:style/:filename"

Paperclip::Attachment.default_options[:fog_credentials] = {
provider: ENV["FOG_PROVIDER"] || "Local",
local_root: "#{Rails.root}/public",
google_project: 'public-lab' ,
google_json_key_location: ENV["GOOGLE_JSON_KEY_FILE"] || ''
}
Paperclip::Attachment.default_options[:fog_host] = ""

# Thank you to http://bendangelo.me/?p=60
module Paperclip
# converts a string into a file for paperclip to save
Expand Down
1 change: 0 additions & 1 deletion config/routes.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,6 @@
get 'spectrums/rss' => 'spectrums#rss'
get 'spectrums/find_brightest_row' => 'spectrums#find_brightest_row'
get 'spectrums/upload' => 'spectrums#upload'
get 'spectrums/setsamplerow' => 'spectrums#setsamplerow'
get 'spectrums/rotate' => 'spectrums#rotate'
get 'spectrums/fork' => 'spectrums#fork'
get 'spectrums/reverse' => 'spectrums#reverse'
Expand Down
5 changes: 5 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ services:
environment:
- PORT=${PORT}
- POOL_SIZE=${POOL_SIZE}
- GOOGLE_STORAGE_SECRET=${GOOGLE_STORAGE_SECRET}
- GOOGLE_STORAGE_KEY=${GOOGLE_STORAGE_KEY}
- GOOGLE_STORAGE_BUCKET_NAME=${GOOGLE_STORAGE_BUCKET_NAME}
- GOOGLE_JSON_KEY_FILE=${GOOGLE_JSON_KEY_FILE}
- FOG_PROVIDER=${FOG_PROVIDER}
ports:
- 127.0.0.1:${PORT:-5000}:${PORT:-5000}
depends_on:
Expand Down