Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2216 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.9884. Min Act: -0.0000

Data Index: 261820 (C4 (Web Text))

Max Activating Token Index: 382

Text #1

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.9781. Min Act: -0.0000

Data Index: 406967 (C4 (Web Text))

Max Activating Token Index: 212

Text #2

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.9675. Min Act: -0.0000

Data Index: 608495 (C4 (Web Text))

Max Activating Token Index: 405

Text #3

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.9565. Min Act: -0.0000

Data Index: 164550 (C4 (Web Text))

Max Activating Token Index: 897

Text #4

Max Range: 4.0993. Min Range: -4.0993

Max Act: 4.0993. Min Act: -0.0000

Data Index: 32296 (C4 (Web Text))

Max Activating Token Index: 758

Text #5

Max Range: 4.0993. Min Range: -4.0993

Max Act: 4.0802. Min Act: -0.0000

Data Index: 206514 (C4 (Web Text))

Max Activating Token Index: 970

Text #6

Max Range: 4.0993. Min Range: -4.0993

Max Act: 4.0036. Min Act: -0.0000

Data Index: 1188019 (C4 (Web Text))

Max Activating Token Index: 249

Text #7

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.7366. Min Act: -0.0000

Data Index: 716751 (C4 (Web Text))

Max Activating Token Index: 730

Text #8

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.9266. Min Act: -0.0000

Data Index: 176558 (C4 (Web Text))

Max Activating Token Index: 627

Text #9

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.6321. Min Act: -0.0000

Data Index: 677407 (C4 (Web Text))

Max Activating Token Index: 477

Text #10

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.6078. Min Act: -0.0000

Data Index: 736022 (C4 (Web Text))

Max Activating Token Index: 740

Text #11

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.6091. Min Act: -0.0000

Data Index: 1136458 (C4 (Web Text))

Max Activating Token Index: 36

Text #12

Max Range: 4.0993. Min Range: -4.0993

Max Act: 3.0568. Min Act: -0.0000

Data Index: 1220147 (C4 (Web Text))

Max Activating Token Index: 111

Text #13

Max Range: 4.0993. Min Range: -4.0993

Max Act: 2.9677. Min Act: -0.0000

Data Index: 251211 (C4 (Web Text))

Max Activating Token Index: 776

Text #14

Max Range: 4.0993. Min Range: -4.0993

Max Act: 2.7666. Min Act: -0.0000

Data Index: 1259794 (C4 (Web Text))

Max Activating Token Index: 279

Text #15

Max Range: 4.0993. Min Range: -4.0993

Max Act: 2.9203. Min Act: -0.0000

Data Index: 537963 (C4 (Web Text))

Max Activating Token Index: 200

Text #16

Max Range: 4.0993. Min Range: -4.0993

Max Act: 2.7248. Min Act: -0.0000

Data Index: 946807 (C4 (Web Text))

Max Activating Token Index: 397

Text #17

Max Range: 4.0993. Min Range: -4.0993

Max Act: 2.9941. Min Act: -0.0000

Data Index: 916119 (C4 (Web Text))

Max Activating Token Index: 401

Text #18

Max Range: 4.0993. Min Range: -4.0993

Max Act: 2.7718. Min Act: -0.0000

Data Index: 801913 (C4 (Web Text))

Max Activating Token Index: 719

Text #19

Max Range: 4.0993. Min Range: -4.0993

Max Act: 2.6263. Min Act: -0.0000

Data Index: 169004 (C4 (Web Text))

Max Activating Token Index: 632