Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2105 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1304. Min Range: -1.1304

Max Act: 1.1075. Min Act: -0.0000

Data Index: 209014 (C4 (Web Text))

Max Activating Token Index: 741

Text #1

Max Range: 1.1304. Min Range: -1.1304

Max Act: 1.1304. Min Act: -0.0000

Data Index: 371577 (C4 (Web Text))

Max Activating Token Index: 330

Text #2

Max Range: 1.1304. Min Range: -1.1304

Max Act: 1.0115. Min Act: -0.0000

Data Index: 1146055 (C4 (Web Text))

Max Activating Token Index: 962

Text #3

Max Range: 1.1304. Min Range: -1.1304

Max Act: 1.0724. Min Act: -0.0000

Data Index: 388115 (C4 (Web Text))

Max Activating Token Index: 166

Text #4

Max Range: 1.1304. Min Range: -1.1304

Max Act: 1.0036. Min Act: -0.0000

Data Index: 777772 (C4 (Web Text))

Max Activating Token Index: 65

Text #5

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9817. Min Act: 0.0001

Data Index: 465090 (C4 (Web Text))

Max Activating Token Index: 432

Text #6

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9683. Min Act: -0.0000

Data Index: 1186297 (C4 (Web Text))

Max Activating Token Index: 573

Text #7

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9003. Min Act: -0.0000

Data Index: 992622 (C4 (Web Text))

Max Activating Token Index: 216

Text #8

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9574. Min Act: -0.0000

Data Index: 504614 (C4 (Web Text))

Max Activating Token Index: 139

Text #9

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9467. Min Act: -0.0000

Data Index: 594251 (C4 (Web Text))

Max Activating Token Index: 491

Text #10

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.8781. Min Act: -0.0000

Data Index: 992888 (C4 (Web Text))

Max Activating Token Index: 809

Text #11

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9126. Min Act: -0.0000

Data Index: 593769 (C4 (Web Text))

Max Activating Token Index: 71

Text #12

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9660. Min Act: 0.0000

Data Index: 550160 (C4 (Web Text))

Max Activating Token Index: 440

Text #13

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9277. Min Act: -0.0000

Data Index: 47909 (C4 (Web Text))

Max Activating Token Index: 414

Text #14

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9469. Min Act: -0.0000

Data Index: 58083 (C4 (Web Text))

Max Activating Token Index: 143

Text #15

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.8627. Min Act: -0.0000

Data Index: 1070018 (C4 (Web Text))

Max Activating Token Index: 654

Text #16

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.9241. Min Act: -0.0000

Data Index: 634968 (C4 (Web Text))

Max Activating Token Index: 280

Text #17

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.8936. Min Act: -0.0000

Data Index: 61134 (C4 (Web Text))

Max Activating Token Index: 477

Text #18

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.8875. Min Act: -0.0000

Data Index: 856498 (C4 (Web Text))

Max Activating Token Index: 738

Text #19

Max Range: 1.1304. Min Range: -1.1304

Max Act: 0.8582. Min Act: -0.0000

Data Index: 692007 (C4 (Web Text))

Max Activating Token Index: 449