Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IP scrubber #2990

Merged
merged 21 commits into from
Sep 5, 2023
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions account/account.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,16 @@ func GetAccount(ctx context.Context, cfg *config.Configuration, fetcher stored_r
return nil, errs
}

errs = account.Privacy.IPv6Config.Validate(errs)
if len(errs) > 0 {
return nil, errs
}
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved

errs = account.Privacy.IPv4Config.Validate(errs)
if len(errs) > 0 {
return nil, errs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reasonable way to add a test that throws this error? Or is it challenging with this Validate function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, added!

}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: Could simplify the syntax...

if errs := account.Privacy.IPv6Config.Validate(nil); len(errs) > 0 {
	account.Privacy.IPv6Config.AnonKeepBits = iputil.IPv6BitSize
}

if errs := account.Privacy.IPv4Config.Validate(nil); len(errs) > 0 {
	account.Privacy.IPv4Config.AnonKeepBits = iputil.IPv4BitSize
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines show as having test coverage, but there doesn't seem to an assertion on setting their default values. I made a type locally where IPv4Config.AnonKeepBits = iputil.IPv6BitSize which is setting the ipv6 value for the ipv4 default and this mistake wasn't caught by a test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is tested in a general GetAccount test.
I added extra flag to assert IP config for account id "invalid_acct_ipv6_ipv4".
I also realized I set both ipv6 and ipv4 to max value, instead of default mask bits. I added 2 new constants:

IPv4DefaultMaskingBitSize = 24
IPv6DefaultMaskingBitSize = 56

For the simplified syntax I added new variables for errors to show explicitly we don't want to add IP config validation errors to errs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also tested this update end to end including test with publisher account id.


// set the value of events.enabled field based on deprecated events_enabled field and ensure backward compatibility
deprecateEventsEnabledField(account)

Expand Down
4 changes: 4 additions & 0 deletions account/account_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package account
import (
"context"
"encoding/json"
"errors"
"fmt"
"testing"

Expand All @@ -19,6 +20,7 @@ import (

var mockAccountData = map[string]json.RawMessage{
"valid_acct": json.RawMessage(`{"disabled":false}`),
"invalid_acct": json.RawMessage(`{"disabled":false, "privacy": {"ipv6": {"anon-keep-bits": -32}}}`),
"disabled_acct": json.RawMessage(`{"disabled":true}`),
"malformed_acct": json.RawMessage(`{"disabled":"invalid type"}`),
"gdpr_channel_enabled_acct": json.RawMessage(`{"disabled":false,"gdpr":{"channel_enabled":{"amp":true}}}`),
Expand Down Expand Up @@ -78,6 +80,8 @@ func TestGetAccount(t *testing.T) {
{accountID: "valid_acct", required: false, disabled: true, err: nil},
{accountID: "valid_acct", required: true, disabled: true, err: nil},

{accountID: "invalid_acct", required: true, disabled: false, err: errors.New("left mask bits cannot exceed 128 in ipv6 address, or be less than 0")},

// pubID given and matches a host account explicitly disabled (Disabled: true on account json)
{accountID: "disabled_acct", required: false, disabled: false, err: &errortypes.BlacklistedAcct{}},
{accountID: "disabled_acct", required: true, disabled: false, err: &errortypes.BlacklistedAcct{}},
Expand Down
37 changes: 35 additions & 2 deletions config/account.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ const (
ChannelWeb ChannelType = "web"
)

const (
IPv4BitSize = 32
IPv6BitSize = 128
)
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved

// Account represents a publisher account configuration
type Account struct {
ID string `mapstructure:"id" json:"id"`
Expand All @@ -40,7 +45,7 @@ type Account struct {
Validations Validations `mapstructure:"validations" json:"validations"`
DefaultBidLimit int `mapstructure:"default_bid_limit" json:"default_bid_limit"`
BidAdjustments *openrtb_ext.ExtRequestPrebidBidAdjustments `mapstructure:"bidadjustments" json:"bidadjustments"`
Privacy *AccountPrivacy `mapstructure:"privacy" json:"privacy"`
Privacy AccountPrivacy `mapstructure:"privacy" json:"privacy"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding, why is this converted to remove its pointer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to non-pointer because we no have a new IPMasking field inside of the Privacy:

type AccountPrivacy struct {
	AllowActivities *AllowActivities `mapstructure:"allowactivities" json:"allowactivities"`
	IPMasking       IPMasking        `mapstructure:"ip_masking" json:"ip_masking"` 
}

We need to have IPMasking to be always present and to be populated with default values from configs.
PBS host account_defaults configs and/or request.account config can overwrite these values.

This is why config.config.go has these lines:

v.SetDefault("account_defaults.privacy.ip_masking.ipv6.left_mask_bits", 56)
v.SetDefault("account_defaults.privacy.ip_masking.ipv4.left_mask_bits", 24)

}

// CookieSync represents the account-level defaults for the cookie sync endpoint.
Expand Down Expand Up @@ -295,5 +300,33 @@ func (a *AccountChannel) IsSet() bool {
}

type AccountPrivacy struct {
AllowActivities AllowActivities `mapstructure:"allowactivities" json:"allowactivities"`
AllowActivities *AllowActivities `mapstructure:"allowactivities" json:"allowactivities"`
IPv6Config IPv6 `mapstructure:"ipv6" json:"ipv6"`
IPv4Config IPv4 `mapstructure:"ipv4" json:"ipv4"`
}

type IPv6 struct {
AnonKeepBits int `mapstructure:"anon-keep-bits" json:"anon-keep-bits"`
// For the follow up PR, always keep first 64 bits
// AlwaysKeepBits int `mapstructure:"always-keep-bits" json:"always-keep-bits"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to confirm that we want to keep this commented code out here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO better to remove the comment and add as a note to the associated issue.

}

type IPv4 struct {
AnonKeepBits int `mapstructure:"anon-keep-bits" json:"anon-keep-bits"`
}
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: Let's use underscores to be consistent with the rest of this object: anon-keep-bits -> anon_keep_bits


func (IPv6Config *IPv6) Validate(errs []error) []error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variables names should start with lower case characters. It's common to use a single character for a function receiver reference. Consider renaming to simply ip?

if IPv6Config.AnonKeepBits > IPv6BitSize || IPv6Config.AnonKeepBits < 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a pointer function receiver, you need to handle the nil case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IPv6 and IPv4 are not pointers in AccountPrivacy:

type AccountPrivacy struct {
	AllowActivities *AllowActivities `mapstructure:"allowactivities" json:"allowactivities"`
	IPv6Config      IPv6             `mapstructure:"ipv6" json:"ipv6"`
	IPv4Config      IPv4             `mapstructure:"ipv4" json:"ipv4"`
}

technically they will be always initialized. Should I add a nil check anyways?

err := fmt.Errorf("left mask bits cannot exceed %d in ipv6 address, or be less than 0", IPv6BitSize)
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved
errs = append(errs, err)
}
return errs
}

func (IPv4Config *IPv4) Validate(errs []error) []error {
if IPv4Config.AnonKeepBits > IPv4BitSize || IPv4Config.AnonKeepBits < 0 {
err := fmt.Errorf("left mask bits cannot exceed %d in ipv4 address, or be less than 0", IPv4BitSize)
errs = append(errs, err)
}
return errs
}
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved
46 changes: 46 additions & 0 deletions config/account_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -910,3 +910,49 @@ func TestAccountPriceFloorsValidate(t *testing.T) {
})
}
}

func TestIPMaskingValidate(t *testing.T) {
tests := []struct {
name string
privacy AccountPrivacy
want []error
}{
{
name: "valid configuration",
privacy: AccountPrivacy{
IPv4Config: IPv4{AnonKeepBits: 1},
IPv6Config: IPv6{AnonKeepBits: 0},
},
},
{
name: "invalid configuration",
privacy: AccountPrivacy{
IPv4Config: IPv4{AnonKeepBits: -100},
IPv6Config: IPv6{AnonKeepBits: -200},
},
want: []error{
errors.New("left mask bits cannot exceed 32 in ipv4 address, or be less than 0"),
errors.New("left mask bits cannot exceed 128 in ipv6 address, or be less than 0"),
},
},
{
name: "mixed valid and invalid configuration",
privacy: AccountPrivacy{
IPv4Config: IPv4{AnonKeepBits: 10},
IPv6Config: IPv6{AnonKeepBits: -10},
},
want: []error{
errors.New("left mask bits cannot exceed 128 in ipv6 address, or be less than 0"),
},
},
SyntaxNode marked this conversation as resolved.
Show resolved Hide resolved
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var errs []error
errs = tt.privacy.IPv4Config.Validate(errs)
errs = tt.privacy.IPv6Config.Validate(errs)
assert.ElementsMatch(t, errs, tt.want)
})
}
}
7 changes: 6 additions & 1 deletion config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,12 +154,15 @@ func (cfg *Configuration) validate(v *viper.Viper) []error {
cfg.TmaxAdjustments.Enabled = false
}

if cfg.AccountDefaults.Privacy != nil {
if cfg.AccountDefaults.Privacy.AllowActivities != nil {
glog.Warning("account_defaults.Privacy has no effect as the feature is under development.")
}

errs = cfg.Experiment.validate(errs)
errs = cfg.BidderInfos.validate(errs)
errs = cfg.AccountDefaults.Privacy.IPv6Config.Validate(errs)
errs = cfg.AccountDefaults.Privacy.IPv4Config.Validate(errs)

return errs
}

Expand Down Expand Up @@ -1024,6 +1027,8 @@ func SetupViper(v *viper.Viper, filename string, bidderInfos BidderInfos) {
v.SetDefault("account_defaults.price_floors.max_rules", 100)
v.SetDefault("account_defaults.price_floors.max_schema_dims", 3)
v.SetDefault("account_defaults.events_enabled", false)
v.SetDefault("account_defaults.privacy.ipv6.anon-keep-bits", 56)
v.SetDefault("account_defaults.privacy.ipv4.anon-keep-bits", 24)

v.SetDefault("compression.response.enable_gzip", false)
v.SetDefault("compression.request.enable_gzip", false)
Expand Down
11 changes: 11 additions & 0 deletions config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,9 @@ func TestDefaults(t *testing.T) {
cmpUnsignedInts(t, "tmax_adjustments.bidder_network_latency_buffer_ms", 0, cfg.TmaxAdjustments.BidderNetworkLatencyBuffer)
cmpUnsignedInts(t, "tmax_adjustments.pbs_response_preparation_duration_ms", 0, cfg.TmaxAdjustments.PBSResponsePreparationDuration)

cmpInts(t, "account_defaults.privacy.ipv6.anon-keep-bits", 56, cfg.AccountDefaults.Privacy.IPv6Config.AnonKeepBits)
cmpInts(t, "account_defaults.privacy.ipv4.anon-keep-bits", 24, cfg.AccountDefaults.Privacy.IPv4Config.AnonKeepBits)

//Assert purpose VendorExceptionMap hash tables were built correctly
expectedTCF2 := TCF2{
Enabled: true,
Expand Down Expand Up @@ -477,6 +480,11 @@ account_defaults:
use_dynamic_data: true
max_rules: 120
max_schema_dims: 5
privacy:
ipv6:
anon-keep-bits: 50
ipv4:
anon-keep-bits: 20
tmax_adjustments:
enabled: true
bidder_response_duration_min_ms: 700
Expand Down Expand Up @@ -583,6 +591,9 @@ func TestFullConfig(t *testing.T) {
cmpBools(t, "account_defaults.events_enabled", *cfg.AccountDefaults.EventsEnabled, true)
cmpNils(t, "account_defaults.events.enabled", cfg.AccountDefaults.Events.Enabled)

cmpInts(t, "account_defaults.privacy.ipv6.anon-keep-bits", 50, cfg.AccountDefaults.Privacy.IPv6Config.AnonKeepBits)
cmpInts(t, "account_defaults.privacy.ipv4.anon-keep-bits", 20, cfg.AccountDefaults.Privacy.IPv4Config.AnonKeepBits)

// Assert compression related defaults
cmpBools(t, "enable_gzip", false, cfg.EnableGzip)
cmpBools(t, "compression.request.enable_gzip", true, cfg.Compression.Request.GZIP)
Expand Down
2 changes: 1 addition & 1 deletion endpoints/cookie_sync.go
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ func (c *cookieSyncEndpoint) parseRequest(r *http.Request) (usersync.Request, pr
}
}

activityControl, activitiesErr := privacy.NewActivityControl(account.Privacy)
activityControl, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
if errortypes.ContainsFatalError([]error{activitiesErr}) {
activityControl = privacy.ActivityControl{}
Expand Down
2 changes: 1 addition & 1 deletion endpoints/cookie_sync_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2108,7 +2108,7 @@ func (p *fakePermissions) AuctionActivitiesAllowed(ctx context.Context, bidderCo

func getDefaultActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
AllowActivities: &config.AllowActivities{
SyncUser: config.Activity{
Default: ptrutil.ToPtr(true),
Rules: []config.ActivityRule{
Expand Down
2 changes: 1 addition & 1 deletion endpoints/openrtb2/amp_auction.go
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ func (deps *endpointDeps) AmpAuction(w http.ResponseWriter, r *http.Request, _ h

tcf2Config := gdpr.NewTCF2Config(deps.cfg.GDPR.TCF2, account.GDPR)

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
errL = append(errL, activitiesErr)
writeError(errL, w, &labels)
Expand Down
2 changes: 1 addition & 1 deletion endpoints/openrtb2/auction.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ func (deps *endpointDeps) Auction(w http.ResponseWriter, r *http.Request, _ http

tcf2Config := gdpr.NewTCF2Config(deps.cfg.GDPR.TCF2, account.GDPR)

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
errL = append(errL, activitiesErr)
if errortypes.ContainsFatalError(errL) {
Expand Down
2 changes: 1 addition & 1 deletion endpoints/openrtb2/video_auction.go
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ func (deps *endpointDeps) VideoAuctionEndpoint(w http.ResponseWriter, r *http.Re
return
}

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
errL = append(errL, activitiesErr)
if errortypes.ContainsFatalError(errL) {
Expand Down
4 changes: 2 additions & 2 deletions endpoints/openrtb2/video_auction_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1299,8 +1299,8 @@ func mockDepsInvalidPrivacy(t *testing.T, ex *mockExchangeVideo) *endpointDeps {
&mockAccountFetcher{data: mockVideoAccountData},
&config.Configuration{MaxRequestSize: maxSize,
AccountDefaults: config.Account{
Privacy: &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
Privacy: config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitPreciseGeo: config.Activity{Rules: []config.ActivityRule{
{Condition: config.ActivityCondition{ComponentName: []string{"bidderA.BidderB.bidderC"}}},
}},
Expand Down
2 changes: 1 addition & 1 deletion endpoints/setuid.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ func NewSetUIDEndpoint(cfg *config.Configuration, syncersByBidder map[string]use
return
}

activities, activitiesErr := privacy.NewActivityControl(account.Privacy)
activities, activitiesErr := privacy.NewActivityControl(&account.Privacy)
if activitiesErr != nil {
if errortypes.ContainsFatalError([]error{activitiesErr}) {
activities = privacy.ActivityControl{}
Expand Down
2 changes: 1 addition & 1 deletion exchange/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ func (rs *requestSplitter) cleanOpenRTBRequests(ctx context.Context,

privacyEnforcement.TID = !auctionReq.Activities.Allow(privacy.ActivityTransmitTids, scopedName)

privacyEnforcement.Apply(bidderRequest.BidRequest)
privacyEnforcement.Apply(bidderRequest.BidRequest, auctionReq.Account.Privacy)
allowedBidderRequests = append(allowedBidderRequests, bidderRequest)

// GPP downgrade: always downgrade unless we can confirm GPP is supported
Expand Down
28 changes: 14 additions & 14 deletions exchange/utils_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4272,7 +4272,7 @@ func TestCleanOpenRTBRequestsActivities(t *testing.T) {
testCases := []struct {
name string
req *openrtb2.BidRequest
privacyConfig *config.AccountPrivacy
privacyConfig config.AccountPrivacy
componentName string
allow bool
expectedReqNumber int
Expand Down Expand Up @@ -4364,7 +4364,7 @@ func TestCleanOpenRTBRequestsActivities(t *testing.T) {
}

for _, test := range testCases {
activities, err := privacy.NewActivityControl(test.privacyConfig)
activities, err := privacy.NewActivityControl(&test.privacyConfig)
assert.NoError(t, err, "")
auctionReq := AuctionRequest{
BidRequestWrapper: &openrtb_ext.RequestWrapper{BidRequest: test.req},
Expand Down Expand Up @@ -4410,33 +4410,33 @@ func buildDefaultActivityConfig(componentName string, allow bool) config.Activit
}
}

func getFetchBidsActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getFetchBidsActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
FetchBids: buildDefaultActivityConfig(componentName, allow),
},
}
}

func getTransmitUFPDActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getTransmitUFPDActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitUserFPD: buildDefaultActivityConfig(componentName, allow),
},
}
}

func getTransmitPreciseGeoActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getTransmitPreciseGeoActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitPreciseGeo: buildDefaultActivityConfig(componentName, allow),
},
}
}

func getTransmitTIDActivityConfig(componentName string, allow bool) *config.AccountPrivacy {
return &config.AccountPrivacy{
AllowActivities: config.AllowActivities{
func getTransmitTIDActivityConfig(componentName string, allow bool) config.AccountPrivacy {
return config.AccountPrivacy{
AllowActivities: &config.AllowActivities{
TransmitTids: buildDefaultActivityConfig(componentName, allow),
},
}
Expand Down
2 changes: 1 addition & 1 deletion privacy/activitycontrol.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ func NewActivityControl(privacyConf *config.AccountPrivacy) (ActivityControl, er
ac := ActivityControl{}
var err error

if privacyConf == nil {
if privacyConf == nil || privacyConf.AllowActivities == nil {
return ac, nil
}

Expand Down
Loading