Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IP scrubber #2990

Merged
merged 21 commits into from
Sep 5, 2023
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion account/account.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ import (
"context"
"encoding/json"
"fmt"

"github.com/buger/jsonparser"
"github.com/prebid/go-gdpr/consentconstants"
"github.com/prebid/prebid-server/config"
"github.com/prebid/prebid-server/errortypes"
"github.com/prebid/prebid-server/metrics"
"github.com/prebid/prebid-server/openrtb_ext"
"github.com/prebid/prebid-server/stored_requests"
"github.com/prebid/prebid-server/util/iputil"
jsonpatch "gopkg.in/evanphx/json-patch.v4"
)

Expand Down Expand Up @@ -103,6 +103,20 @@ func GetAccount(ctx context.Context, cfg *config.Configuration, fetcher stored_r
return nil, errs
}

var ipv6Errs []error
ipv6Errs = account.Privacy.IPv6Config.Validate(ipv6Errs)
if len(ipv6Errs) > 0 {
account.Privacy.IPv6Config.AnonKeepBits = iputil.IPv6BitSize
// record invalid ipv6 metric
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you commented asking if this can be done in a separate PR, if so can this comment be removed?

}
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved

var ipv4Errs []error
ipv4Errs = account.Privacy.IPv4Config.Validate(ipv4Errs)
if len(ipv4Errs) > 0 {
account.Privacy.IPv4Config.AnonKeepBits = iputil.IPv4BitSize
// record invalid ipv4 metric
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same point about removing metric related comment

}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: Could simplify the syntax...

if errs := account.Privacy.IPv6Config.Validate(nil); len(errs) > 0 {
	account.Privacy.IPv6Config.AnonKeepBits = iputil.IPv6BitSize
}

if errs := account.Privacy.IPv4Config.Validate(nil); len(errs) > 0 {
	account.Privacy.IPv4Config.AnonKeepBits = iputil.IPv4BitSize
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines show as having test coverage, but there doesn't seem to an assertion on setting their default values. I made a type locally where IPv4Config.AnonKeepBits = iputil.IPv6BitSize which is setting the ipv6 value for the ipv4 default and this mistake wasn't caught by a test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is tested in a general GetAccount test.
I added extra flag to assert IP config for account id "invalid_acct_ipv6_ipv4".
I also realized I set both ipv6 and ipv4 to max value, instead of default mask bits. I added 2 new constants:

IPv4DefaultMaskingBitSize = 24
IPv6DefaultMaskingBitSize = 56

For the simplified syntax I added new variables for errors to show explicitly we don't want to add IP config validation errors to errs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also tested this update end to end including test with publisher account id.


// set the value of events.enabled field based on deprecated events_enabled field and ensure backward compatibility
deprecateEventsEnabledField(account)

Expand Down
15 changes: 10 additions & 5 deletions account/account_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@ import (
)

var mockAccountData = map[string]json.RawMessage{
"valid_acct": json.RawMessage(`{"disabled":false}`),
"disabled_acct": json.RawMessage(`{"disabled":true}`),
"malformed_acct": json.RawMessage(`{"disabled":"invalid type"}`),
"gdpr_channel_enabled_acct": json.RawMessage(`{"disabled":false,"gdpr":{"channel_enabled":{"amp":true}}}`),
"ccpa_channel_enabled_acct": json.RawMessage(`{"disabled":false,"ccpa":{"channel_enabled":{"amp":true}}}`),
"valid_acct": json.RawMessage(`{"disabled":false}`),
"invalid_acct_ipv6": json.RawMessage(`{"disabled":false, "privacy": {"ipv6": {"anon-keep-bits": -32}, "ipv4": {"anon-keep-bits": 16}}}`),
"invalid_acct_ipv4": json.RawMessage(`{"disabled":false, "privacy": {"ipv4": {"anon-keep-bits": -32}, "ipv6": {"anon-keep-bits": 32}}}`),
"disabled_acct": json.RawMessage(`{"disabled":true}`),
"malformed_acct": json.RawMessage(`{"disabled":"invalid type"}`),
"gdpr_channel_enabled_acct": json.RawMessage(`{"disabled":false,"gdpr":{"channel_enabled":{"amp":true}}}`),
"ccpa_channel_enabled_acct": json.RawMessage(`{"disabled":false,"ccpa":{"channel_enabled":{"amp":true}}}`),
"gdpr_channel_enabled_deprecated_purpose_acct": json.RawMessage(`{"disabled":false,"gdpr":{"purpose1":{"enforce_purpose":"full"}, "channel_enabled":{"amp":true}}}`),
"gdpr_deprecated_purpose1": json.RawMessage(`{"disabled":false,"gdpr":{"purpose1":{"enforce_purpose":"full"}}}`),
"gdpr_deprecated_purpose2": json.RawMessage(`{"disabled":false,"gdpr":{"purpose2":{"enforce_purpose":"full"}}}`),
Expand Down Expand Up @@ -78,6 +80,9 @@ func TestGetAccount(t *testing.T) {
{accountID: "valid_acct", required: false, disabled: true, err: nil},
{accountID: "valid_acct", required: true, disabled: true, err: nil},

{accountID: "invalid_acct_ipv6", required: true, disabled: false, err: nil},
{accountID: "invalid_acct_ipv4", required: true, disabled: false, err: nil},

// pubID given and matches a host account explicitly disabled (Disabled: true on account json)
{accountID: "disabled_acct", required: false, disabled: false, err: &errortypes.BlacklistedAcct{}},
{accountID: "disabled_acct", required: true, disabled: false, err: &errortypes.BlacklistedAcct{}},
Expand Down
31 changes: 29 additions & 2 deletions config/account.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (

"github.com/prebid/go-gdpr/consentconstants"
"github.com/prebid/prebid-server/openrtb_ext"
"github.com/prebid/prebid-server/util/iputil"
)

// ChannelType enumerates the values of integrations Prebid Server can configure for an account
Expand Down Expand Up @@ -40,7 +41,7 @@ type Account struct {
Validations Validations `mapstructure:"validations" json:"validations"`
DefaultBidLimit int `mapstructure:"default_bid_limit" json:"default_bid_limit"`
BidAdjustments *openrtb_ext.ExtRequestPrebidBidAdjustments `mapstructure:"bidadjustments" json:"bidadjustments"`
Privacy *AccountPrivacy `mapstructure:"privacy" json:"privacy"`
Privacy AccountPrivacy `mapstructure:"privacy" json:"privacy"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding, why is this converted to remove its pointer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to non-pointer because we no have a new IPMasking field inside of the Privacy:

type AccountPrivacy struct {
	AllowActivities *AllowActivities `mapstructure:"allowactivities" json:"allowactivities"`
	IPMasking       IPMasking        `mapstructure:"ip_masking" json:"ip_masking"` 
}

We need to have IPMasking to be always present and to be populated with default values from configs.
PBS host account_defaults configs and/or request.account config can overwrite these values.

This is why config.config.go has these lines:

v.SetDefault("account_defaults.privacy.ip_masking.ipv6.left_mask_bits", 56)
v.SetDefault("account_defaults.privacy.ip_masking.ipv4.left_mask_bits", 24)

}

// CookieSync represents the account-level defaults for the cookie sync endpoint.
Expand Down Expand Up @@ -295,5 +296,31 @@ func (a *AccountChannel) IsSet() bool {
}

type AccountPrivacy struct {
AllowActivities AllowActivities `mapstructure:"allowactivities" json:"allowactivities"`
AllowActivities *AllowActivities `mapstructure:"allowactivities" json:"allowactivities"`
IPv6Config IPv6 `mapstructure:"ipv6" json:"ipv6"`
IPv4Config IPv4 `mapstructure:"ipv4" json:"ipv4"`
}

type IPv6 struct {
AnonKeepBits int `mapstructure:"anon-keep-bits" json:"anon-keep-bits"`
}

type IPv4 struct {
AnonKeepBits int `mapstructure:"anon-keep-bits" json:"anon-keep-bits"`
}
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: Let's use underscores to be consistent with the rest of this object: anon-keep-bits -> anon_keep_bits


func (ip *IPv6) Validate(errs []error) []error {
if ip.AnonKeepBits > iputil.IPv6BitSize || ip.AnonKeepBits < 0 {
err := fmt.Errorf("bits cannot exceed %d in ipv6 address, or be less than 0", iputil.IPv6BitSize)
errs = append(errs, err)
}
return errs
}

func (ip *IPv4) Validate(errs []error) []error {
if ip.AnonKeepBits > iputil.IPv4BitSize || ip.AnonKeepBits < 0 {
err := fmt.Errorf("bits cannot exceed %d in ipv4 address, or be less than 0", iputil.IPv4BitSize)
errs = append(errs, err)
}
return errs
}
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved
46 changes: 46 additions & 0 deletions config/account_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -910,3 +910,49 @@ func TestAccountPriceFloorsValidate(t *testing.T) {
})
}
}

func TestIPMaskingValidate(t *testing.T) {
tests := []struct {
name string
privacy AccountPrivacy
want []error
}{
{
name: "valid",
privacy: AccountPrivacy{
IPv4Config: IPv4{AnonKeepBits: 1},
IPv6Config: IPv6{AnonKeepBits: 0},
},
},
{
name: "invalid",
privacy: AccountPrivacy{
IPv4Config: IPv4{AnonKeepBits: -100},
IPv6Config: IPv6{AnonKeepBits: -200},
},
want: []error{
errors.New("bits cannot exceed 32 in ipv4 address, or be less than 0"),
errors.New("bits cannot exceed 128 in ipv6 address, or be less than 0"),
},
},
{
name: "mixed",
privacy: AccountPrivacy{
IPv4Config: IPv4{AnonKeepBits: 10},
IPv6Config: IPv6{AnonKeepBits: -10},
},
want: []error{
errors.New("bits cannot exceed 128 in ipv6 address, or be less than 0"),
},
},
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var errs []error
errs = tt.privacy.IPv4Config.Validate(errs)
errs = tt.privacy.IPv6Config.Validate(errs)
assert.ElementsMatch(t, errs, tt.want)
})
}
}
9 changes: 5 additions & 4 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -149,12 +149,11 @@ func (cfg *Configuration) validate(v *viper.Viper) []error {
errs = append(errs, errors.New("account_defaults.Events.VASTEvents has no effect as the feature is under development."))
}

if cfg.AccountDefaults.Privacy != nil {
glog.Warning("account_defaults.Privacy has no effect as the feature is under development.")
}

errs = cfg.Experiment.validate(errs)
errs = cfg.BidderInfos.validate(errs)
errs = cfg.AccountDefaults.Privacy.IPv6Config.Validate(errs)
errs = cfg.AccountDefaults.Privacy.IPv4Config.Validate(errs)

return errs
}

Expand Down Expand Up @@ -1019,6 +1018,8 @@ func SetupViper(v *viper.Viper, filename string, bidderInfos BidderInfos) {
v.SetDefault("account_defaults.price_floors.max_rules", 100)
v.SetDefault("account_defaults.price_floors.max_schema_dims", 3)
v.SetDefault("account_defaults.events_enabled", false)
v.SetDefault("account_defaults.privacy.ipv6.anon-keep-bits", 56)
v.SetDefault("account_defaults.privacy.ipv4.anon-keep-bits", 24)

v.SetDefault("compression.response.enable_gzip", false)
v.SetDefault("compression.request.enable_gzip", false)
Expand Down
11 changes: 11 additions & 0 deletions config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,9 @@ func TestDefaults(t *testing.T) {
cmpUnsignedInts(t, "tmax_adjustments.bidder_network_latency_buffer_ms", 0, cfg.TmaxAdjustments.BidderNetworkLatencyBuffer)
cmpUnsignedInts(t, "tmax_adjustments.pbs_response_preparation_duration_ms", 0, cfg.TmaxAdjustments.PBSResponsePreparationDuration)

cmpInts(t, "account_defaults.privacy.ipv6.anon-keep-bits", 56, cfg.AccountDefaults.Privacy.IPv6Config.AnonKeepBits)
cmpInts(t, "account_defaults.privacy.ipv4.anon-keep-bits", 24, cfg.AccountDefaults.Privacy.IPv4Config.AnonKeepBits)

//Assert purpose VendorExceptionMap hash tables were built correctly
expectedTCF2 := TCF2{
Enabled: true,
Expand Down Expand Up @@ -477,6 +480,11 @@ account_defaults:
use_dynamic_data: true
max_rules: 120
max_schema_dims: 5
privacy:
ipv6:
anon-keep-bits: 50
ipv4:
anon-keep-bits: 20
tmax_adjustments:
enabled: true
bidder_response_duration_min_ms: 700
Expand Down Expand Up @@ -583,6 +591,9 @@ func TestFullConfig(t *testing.T) {
cmpBools(t, "account_defaults.events_enabled", *cfg.AccountDefaults.EventsEnabled, true)
cmpNils(t, "account_defaults.events.enabled", cfg.AccountDefaults.Events.Enabled)

cmpInts(t, "account_defaults.privacy.ipv6.anon-keep-bits", 50, cfg.AccountDefaults.Privacy.IPv6Config.AnonKeepBits)
cmpInts(t, "account_defaults.privacy.ipv4.anon-keep-bits", 20, cfg.AccountDefaults.Privacy.IPv4Config.AnonKeepBits)

// Assert compression related defaults
cmpBools(t, "enable_gzip", false, cfg.EnableGzip)
cmpBools(t, "compression.request.enable_gzip", true, cfg.Compression.Request.GZIP)
Expand Down
2 changes: 1 addition & 1 deletion endpoints/cookie_sync.go
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ func (c *cookieSyncEndpoint) parseRequest(r *http.Request) (usersync.Request, pr
}
}

activityControl, activitiesErr := privacy.NewActivityControl(account.Privacy)
activityControl, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
if errortypes.ContainsFatalError([]error{activitiesErr}) {
activityControl = privacy.ActivityControl{}
Expand Down
2 changes: 1 addition & 1 deletion endpoints/cookie_sync_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2108,7 +2108,7 @@ func (p *fakePermissions) AuctionActivitiesAllowed(ctx context.Context, bidderCo

func getDefaultActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
AllowActivities: &config.AllowActivities{
SyncUser: config.Activity{
Default: ptrutil.ToPtr(true),
Rules: []config.ActivityRule{
Expand Down
2 changes: 1 addition & 1 deletion endpoints/openrtb2/amp_auction.go
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ func (deps *endpointDeps) AmpAuction(w http.ResponseWriter, r *http.Request, _ h

tcf2Config := gdpr.NewTCF2Config(deps.cfg.GDPR.TCF2, account.GDPR)

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
errL = append(errL, activitiesErr)
writeError(errL, w, &labels)
Expand Down
2 changes: 1 addition & 1 deletion endpoints/openrtb2/auction.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ func (deps *endpointDeps) Auction(w http.ResponseWriter, r *http.Request, _ http

tcf2Config := gdpr.NewTCF2Config(deps.cfg.GDPR.TCF2, account.GDPR)

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
errL = append(errL, activitiesErr)
if errortypes.ContainsFatalError(errL) {
Expand Down
2 changes: 1 addition & 1 deletion endpoints/openrtb2/video_auction.go
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ func (deps *endpointDeps) VideoAuctionEndpoint(w http.ResponseWriter, r *http.Re
return
}

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
errL = append(errL, activitiesErr)
if errortypes.ContainsFatalError(errL) {
Expand Down
4 changes: 2 additions & 2 deletions endpoints/openrtb2/video_auction_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1299,8 +1299,8 @@ func mockDepsInvalidPrivacy(t *testing.T, ex *mockExchangeVideo) *endpointDeps {
&mockAccountFetcher{data: mockVideoAccountData},
&config.Configuration{MaxRequestSize: maxSize,
AccountDefaults: config.Account{
Privacy: &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
Privacy: config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitPreciseGeo: config.Activity{Rules: []config.ActivityRule{
{Condition: config.ActivityCondition{ComponentName: []string{"bidderA.BidderB.bidderC"}}},
}},
Expand Down
2 changes: 1 addition & 1 deletion endpoints/setuid.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ func NewSetUIDEndpoint(cfg *config.Configuration, syncersByBidder map[string]use
return
}

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
if errortypes.ContainsFatalError([]error{activitiesErr}) {
activities = privacy.ActivityControl{}
Expand Down
2 changes: 1 addition & 1 deletion exchange/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ func (rs *requestSplitter) cleanOpenRTBRequests(ctx context.Context,

privacyEnforcement.TID = !auctionReq.Activities.Allow(privacy.ActivityTransmitTids, scopedName)

privacyEnforcement.Apply(bidderRequest.BidRequest)
privacyEnforcement.Apply(bidderRequest.BidRequest, auctionReq.Account.Privacy)
allowedBidderRequests = append(allowedBidderRequests, bidderRequest)

// GPP downgrade: always downgrade unless we can confirm GPP is supported
Expand Down
28 changes: 14 additions & 14 deletions exchange/utils_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4272,7 +4272,7 @@ func TestCleanOpenRTBRequestsActivities(t *testing.T) {
testCases := []struct {
name string
req *openrtb2.BidRequest
privacyConfig *config.AccountPrivacy
privacyConfig config.AccountPrivacy
componentName string
allow bool
expectedReqNumber int
Expand Down Expand Up @@ -4364,7 +4364,7 @@ func TestCleanOpenRTBRequestsActivities(t *testing.T) {
}

for _, test := range testCases {
activities, err := privacy.NewActivityControl(test.privacyConfig)
activities, err := privacy.NewActivityControl(&test.privacyConfig)
assert.NoError(t, err, "")
auctionReq := AuctionRequest{
BidRequestWrapper: &openrtb_ext.RequestWrapper{BidRequest: test.req},
Expand Down Expand Up @@ -4410,33 +4410,33 @@ func buildDefaultActivityConfig(componentName string, allow bool) config.Activit
}
}

func getFetchBidsActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getFetchBidsActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
FetchBids: buildDefaultActivityConfig(componentName, allow),
},
}
}

func getTransmitUFPDActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getTransmitUFPDActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitUserFPD: buildDefaultActivityConfig(componentName, allow),
},
}
}

func getTransmitPreciseGeoActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getTransmitPreciseGeoActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitPreciseGeo: buildDefaultActivityConfig(componentName, allow),
},
}
}

func getTransmitTIDActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getTransmitTIDActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitTids: buildDefaultActivityConfig(componentName, allow),
},
}
Expand Down
2 changes: 1 addition & 1 deletion privacy/activitycontrol.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ func NewActivityControl(privacyConf *config.AccountPrivacy) (ActivityControl, er
ac := ActivityControl{}
var err error

if privacyConf == nil {
if privacyConf == nil || privacyConf.AllowActivities == nil {
return ac, nil
}

Expand Down
Loading