Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1329 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.9919. Min Act: -0.0000

Data Index: 1298625 (C4 (Web Text))

Max Activating Token Index: 244

Text #1

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.9469. Min Act: -0.0000

Data Index: 386434 (C4 (Web Text))

Max Activating Token Index: 986

Text #2

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.7884. Min Act: -0.0000

Data Index: 21695 (C4 (Web Text))

Max Activating Token Index: 497

Text #3

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.7679. Min Act: -0.0000

Data Index: 1263013 (C4 (Web Text))

Max Activating Token Index: 639

Text #4

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.6889. Min Act: 0.0001

Data Index: 315670 (C4 (Web Text))

Max Activating Token Index: 139

Text #5

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.7805. Min Act: -0.0000

Data Index: 1120652 (C4 (Web Text))

Max Activating Token Index: 708

Text #6

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.5354. Min Act: -0.0000

Data Index: 1308283 (C4 (Web Text))

Max Activating Token Index: 692

Text #7

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.5938. Min Act: -0.0000

Data Index: 597841 (C4 (Web Text))

Max Activating Token Index: 485

Text #8

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.4436. Min Act: -0.0000

Data Index: 1257761 (C4 (Web Text))

Max Activating Token Index: 151

Text #9

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.5292. Min Act: -0.0000

Data Index: 273944 (C4 (Web Text))

Max Activating Token Index: 578

Text #10

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.4393. Min Act: 0.0000

Data Index: 1167329 (C4 (Web Text))

Max Activating Token Index: 340

Text #11

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.4867. Min Act: 0.0000

Data Index: 1193411 (C4 (Web Text))

Max Activating Token Index: 365

Text #12

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.4605. Min Act: -0.0000

Data Index: 670734 (C4 (Web Text))

Max Activating Token Index: 904

Text #13

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.4328. Min Act: 0.0000

Data Index: 440860 (C4 (Web Text))

Max Activating Token Index: 885

Text #14

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.3787. Min Act: -0.0000

Data Index: 496670 (C4 (Web Text))

Max Activating Token Index: 732

Text #15

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.3318. Min Act: 0.0000

Data Index: 324516 (C4 (Web Text))

Max Activating Token Index: 232

Text #16

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.1567. Min Act: -0.0000

Data Index: 701259 (C4 (Web Text))

Max Activating Token Index: 414

Text #17

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.2985. Min Act: -0.0000

Data Index: 88937 (C4 (Web Text))

Max Activating Token Index: 750

Text #18

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.2030. Min Act: -0.0000

Data Index: 329487 (C4 (Web Text))

Max Activating Token Index: 296

Text #19

Max Range: 1.9919. Min Range: -1.9919

Max Act: 1.2932. Min Act: -0.0000

Data Index: 328923 (C4 (Web Text))

Max Activating Token Index: 974