Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.1128. Min Act: -0.0003

Data Index: 1175102 (C4 (Web Text))

Max Activating Token Index: 526

Text #1

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.1026. Min Act: -0.0003

Data Index: 422894 (C4 (Web Text))

Max Activating Token Index: 1001

Text #2

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0917. Min Act: -0.0003

Data Index: 777380 (C4 (Web Text))

Max Activating Token Index: 853

Text #3

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0905. Min Act: -0.0003

Data Index: 1068324 (C4 (Web Text))

Max Activating Token Index: 229

Text #4

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0883. Min Act: -0.0003

Data Index: 634107 (C4 (Web Text))

Max Activating Token Index: 464

Text #5

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0832. Min Act: -0.0003

Data Index: 452524 (C4 (Web Text))

Max Activating Token Index: 952

Text #6

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0886. Min Act: -0.0003

Data Index: 310088 (C4 (Web Text))

Max Activating Token Index: 304

Text #7

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0830. Min Act: -0.0003

Data Index: 778210 (C4 (Web Text))

Max Activating Token Index: 696

Text #8

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0806. Min Act: -0.0003

Data Index: 1294271 (C4 (Web Text))

Max Activating Token Index: 573

Text #9

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0809. Min Act: -0.0003

Data Index: 1157562 (C4 (Web Text))

Max Activating Token Index: 466

Text #10

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0758. Min Act: -0.0003

Data Index: 892623 (C4 (Web Text))

Max Activating Token Index: 882

Text #11

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0748. Min Act: -0.0003

Data Index: 1530117 (Python Code)

Max Activating Token Index: 289

Text #12

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0748. Min Act: -0.0003

Data Index: 1361891 (Python Code)

Max Activating Token Index: 141

Text #13

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0730. Min Act: -0.0003

Data Index: 1401788 (Python Code)

Max Activating Token Index: 259

Text #14

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0770. Min Act: -0.0003

Data Index: 1517875 (Python Code)

Max Activating Token Index: 334

Text #15

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0723. Min Act: -0.0003

Data Index: 411257 (C4 (Web Text))

Max Activating Token Index: 341

Text #16

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0726. Min Act: -0.0003

Data Index: 1410631 (Python Code)

Max Activating Token Index: 122

Text #17

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0717. Min Act: -0.0003

Data Index: 586831 (C4 (Web Text))

Max Activating Token Index: 496

Text #18

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0709. Min Act: -0.0003

Data Index: 1590609 (Python Code)

Max Activating Token Index: 185

Text #19

Max Range: 0.1128. Min Range: -0.1128

Max Act: 0.0736. Min Act: -0.0003

Data Index: 1494954 (Python Code)

Max Activating Token Index: 194