Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3096 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4908. Min Act: -0.0000

Data Index: 115057 (C4 (Web Text))

Max Activating Token Index: 253

Text #1

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4706. Min Act: -0.0000

Data Index: 461146 (C4 (Web Text))

Max Activating Token Index: 764

Text #2

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4506. Min Act: -0.0000

Data Index: 451787 (C4 (Web Text))

Max Activating Token Index: 769

Text #3

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4654. Min Act: -0.0000

Data Index: 367044 (C4 (Web Text))

Max Activating Token Index: 55

Text #4

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4281. Min Act: -0.0000

Data Index: 288420 (C4 (Web Text))

Max Activating Token Index: 152

Text #5

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4406. Min Act: -0.0000

Data Index: 1004612 (C4 (Web Text))

Max Activating Token Index: 139

Text #6

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4296. Min Act: -0.0000

Data Index: 72076 (C4 (Web Text))

Max Activating Token Index: 673

Text #7

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3996. Min Act: -0.0000

Data Index: 172722 (C4 (Web Text))

Max Activating Token Index: 258

Text #8

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4083. Min Act: -0.0000

Data Index: 804495 (C4 (Web Text))

Max Activating Token Index: 79

Text #9

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4244. Min Act: -0.0000

Data Index: 274067 (C4 (Web Text))

Max Activating Token Index: 792

Text #10

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4193. Min Act: -0.0000

Data Index: 89236 (C4 (Web Text))

Max Activating Token Index: 958

Text #11

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.4023. Min Act: -0.0000

Data Index: 1030952 (C4 (Web Text))

Max Activating Token Index: 104

Text #12

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3947. Min Act: -0.0000

Data Index: 796310 (C4 (Web Text))

Max Activating Token Index: 688

Text #13

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3698. Min Act: -0.0000

Data Index: 972937 (C4 (Web Text))

Max Activating Token Index: 854

Text #14

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3910. Min Act: -0.0000

Data Index: 326282 (C4 (Web Text))

Max Activating Token Index: 97

Text #15

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3855. Min Act: -0.0000

Data Index: 1045669 (C4 (Web Text))

Max Activating Token Index: 756

Text #16

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3998. Min Act: -0.0000

Data Index: 531306 (C4 (Web Text))

Max Activating Token Index: 30

Text #17

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3856. Min Act: -0.0000

Data Index: 1358590 (C4 (Web Text))

Max Activating Token Index: 356

Text #18

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3603. Min Act: -0.0000

Data Index: 180717 (C4 (Web Text))

Max Activating Token Index: 49

Text #19

Max Range: 0.4908. Min Range: -0.4908

Max Act: 0.3965. Min Act: -0.0000

Data Index: 588077 (C4 (Web Text))

Max Activating Token Index: 997