Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 763 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.4717. Min Act: -0.0000

Data Index: 440645 (C4 (Web Text))

Max Activating Token Index: 1020

Text #1

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3824. Min Act: -0.0000

Data Index: 457149 (C4 (Web Text))

Max Activating Token Index: 342

Text #2

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3666. Min Act: -0.0000

Data Index: 376884 (C4 (Web Text))

Max Activating Token Index: 109

Text #3

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3603. Min Act: 0.0000

Data Index: 1243322 (C4 (Web Text))

Max Activating Token Index: 580

Text #4

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3295. Min Act: -0.0000

Data Index: 160934 (C4 (Web Text))

Max Activating Token Index: 920

Text #5

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3364. Min Act: -0.0000

Data Index: 947579 (C4 (Web Text))

Max Activating Token Index: 884

Text #6

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3470. Min Act: -0.0000

Data Index: 180682 (C4 (Web Text))

Max Activating Token Index: 295

Text #7

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3288. Min Act: -0.0000

Data Index: 752747 (C4 (Web Text))

Max Activating Token Index: 237

Text #8

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3121. Min Act: -0.0000

Data Index: 1012654 (C4 (Web Text))

Max Activating Token Index: 724

Text #9

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3287. Min Act: -0.0000

Data Index: 380113 (C4 (Web Text))

Max Activating Token Index: 960

Text #10

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2958. Min Act: -0.0000

Data Index: 70674 (C4 (Web Text))

Max Activating Token Index: 268

Text #11

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2966. Min Act: -0.0000

Data Index: 1316517 (C4 (Web Text))

Max Activating Token Index: 423

Text #12

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.3155. Min Act: -0.0000

Data Index: 257092 (C4 (Web Text))

Max Activating Token Index: 533

Text #13

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2751. Min Act: -0.0000

Data Index: 676539 (C4 (Web Text))

Max Activating Token Index: 320

Text #14

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2934. Min Act: -0.0000

Data Index: 496134 (C4 (Web Text))

Max Activating Token Index: 838

Text #15

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2838. Min Act: -0.0000

Data Index: 1126074 (C4 (Web Text))

Max Activating Token Index: 776

Text #16

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2955. Min Act: 0.0000

Data Index: 1091605 (C4 (Web Text))

Max Activating Token Index: 40

Text #17

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2776. Min Act: -0.0000

Data Index: 459665 (C4 (Web Text))

Max Activating Token Index: 140

Text #18

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2794. Min Act: -0.0000

Data Index: 700568 (C4 (Web Text))

Max Activating Token Index: 991

Text #19

Max Range: 0.4717. Min Range: -0.4717

Max Act: 0.2757. Min Act: -0.0000

Data Index: 1235038 (C4 (Web Text))

Max Activating Token Index: 467