Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2913 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.5464. Min Act: -0.0001

Data Index: 524939 (C4 (Web Text))

Max Activating Token Index: 421

Text #1

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.5858. Min Act: -0.0001

Data Index: 187947 (C4 (Web Text))

Max Activating Token Index: 253

Text #2

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.5628. Min Act: -0.0001

Data Index: 391609 (C4 (Web Text))

Max Activating Token Index: 993

Text #3

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.5430. Min Act: -0.0001

Data Index: 942 (C4 (Web Text))

Max Activating Token Index: 1012

Text #4

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.5245. Min Act: -0.0001

Data Index: 1043830 (C4 (Web Text))

Max Activating Token Index: 508

Text #5

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.5390. Min Act: -0.0001

Data Index: 459122 (C4 (Web Text))

Max Activating Token Index: 916

Text #6

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4963. Min Act: -0.0001

Data Index: 873308 (C4 (Web Text))

Max Activating Token Index: 976

Text #7

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4767. Min Act: -0.0001

Data Index: 349461 (C4 (Web Text))

Max Activating Token Index: 234

Text #8

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4822. Min Act: -0.0001

Data Index: 287536 (C4 (Web Text))

Max Activating Token Index: 647

Text #9

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4918. Min Act: -0.0001

Data Index: 230371 (C4 (Web Text))

Max Activating Token Index: 380

Text #10

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.5076. Min Act: -0.0001

Data Index: 61856 (C4 (Web Text))

Max Activating Token Index: 138

Text #11

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4860. Min Act: -0.0001

Data Index: 73197 (C4 (Web Text))

Max Activating Token Index: 443

Text #12

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4892. Min Act: -0.0001

Data Index: 985165 (C4 (Web Text))

Max Activating Token Index: 988

Text #13

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4477. Min Act: -0.0001

Data Index: 1005911 (C4 (Web Text))

Max Activating Token Index: 349

Text #14

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4854. Min Act: -0.0001

Data Index: 391875 (C4 (Web Text))

Max Activating Token Index: 357

Text #15

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4897. Min Act: -0.0001

Data Index: 463727 (C4 (Web Text))

Max Activating Token Index: 753

Text #16

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4670. Min Act: -0.0001

Data Index: 889002 (C4 (Web Text))

Max Activating Token Index: 227

Text #17

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4663. Min Act: -0.0001

Data Index: 261014 (C4 (Web Text))

Max Activating Token Index: 495

Text #18

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4789. Min Act: -0.0001

Data Index: 822422 (C4 (Web Text))

Max Activating Token Index: 738

Text #19

Max Range: 0.5858. Min Range: -0.5858

Max Act: 0.4457. Min Act: -0.0001

Data Index: 928358 (C4 (Web Text))

Max Activating Token Index: 262